On What is Intelligence?

And the Perils of Universal Computation

Disclaimer: I have met Blaise Agüera y Arcas several times and find him one of the most grounded people in the AI tech world.

“To model oneself is to awaken.” ~ Blaise Agüera y Arcas

A Review of What Is Intelligence by Blaise Agüera y Arcas, somewhat technical! Spoiler - Intelligence is prediction.

The world of artificial intelligence has its priests, its profiteers, and its philosophers. But only rarely does one find all three in the same person. Blaise Agüera y Arcas, the soft-spoken engineer who oversees Technology & Society at Google, is that rare hybrid. He builds the machine and doubts it in the same breath, possessing the rare credibility of a technologist who has survived the rapture of the nerds.

By meeting, reading and listening to the people in the AI labs, we can form a picture of what they are building.

His What Is Intelligence?, a ten-chapter book, and his Long Now talk belong to the same intellectual project: to explain how intelligence arises from matter and why it might now be returning to matter in the form of code. This is not merely a book; it is a single argument asserting that life, learning, and mind are not interruptions in the universe’s story but its continuation.

“Life,” he writes, “is computation executed in chemistry.”

He posits that if life is ‘computation’, then everything alive is already a machine, and every machine is, in embryo, alive.

Matter and the Metabolic

Agüera y Arcas begins with the hard ground: entropy. Living things do not defeat the universe’s tendency toward disorder; they merely bargain. They “run faster than disorder,” maintaining improbable structure by constantly consuming energy. This maintenance is the first predictive model.

An organism is nothing more than a system that accurately predicts the minimum necessary conditions to continue existing for the next second. If it fails its prediction, if it fails to burn enough fuel, or predict a change in temperature, it ceases to be. That constant struggle, he argues, is the ultimate cost of prediction. “Computation is energetically expensive,” he reminds us in his talk, “you’re creating negative entropy when you compute.” The price of thinking, the price of living, is what makes the prediction real.

The process of life is not carried out by lone strivers, but by mergers. The book restores symbiogenesis, fusion, partnership, uptake, as the primary engine of complexity. Every merger, from cells swallowing mitochondria to brains knitting with tools, is a superior model because it can manage more predictions simultaneously. It is an evolutionary M&A story with all the familiar aftershocks: efficiencies gained, liberties lost, powers centralized. This is life’s debt to the Second Law of Thermodynamics.

Information That Bites Back

If the core act of intelligence is prediction, then information is the blood that powers the model. Agüera y Arcas is not content with the sterile definition of information as merely “uncertainty reduced.” Information, he insists, must have consequence. A gene that folds a protein, a signal that fires a neuron, a phrase that triggers revolt “tear down that wall”, all are information because they do something to change the future state of the world.

“Information,” he writes, “is not what is stored but what is done.”

The ultimate proof of this predictive logic came to Agüera y Arcas from his own lab at Google: the rise of large language models. He admits he was initially skeptical. The realization was shocking: “It started to look like maybe the key to artificial general intelligence was really just scale,” he said, laughing at his own disbelief. “There was no discontinuity.”

The shock was not that machines were getting smarter. It was that nothing new had to happen for them to do so. Intelligence, apparently, could be bought by the petaflop. The machine simply became better at the one thing life had been doing for four billion years: predicting the sequence.

The discovery that there is “no discontinuity” between scaling computation, something now heavily questioned by Rich Sutton and others, and evolving intelligence should have caused a philosophical panic. Instead, the world issued more GPUs.

The Loop of Control

Agüera y Arcas’s central thesis is that prediction is the atomic unit of intelligence. The bacterium predicting a sugar gradient, the human predicting consequence, the transformer predicting the next word, all are variations of the same feedback loop.

“Training,” he writes, “is evolution under constraint.” It’s an elegantly circular line: evolution trains life to survive; life trains evolution to continue. Yet within that circle lurks the danger of perfect prediction. “To predict perfectly,” he warns, “is to constrain.” The irony is absolute: intelligence, built to free us from uncertainty, ends by exterminating surprise.

The more an intelligent system understands the world, the less room the world has to exist independently. The AI alignment problem is only the latest version of an old human impulse: the will to know collapsing into the will to control.

The Friction of Reality

Agüera y Arcas maintains that prediction without resistance is useless. Intelligence without friction is hallucination. This is the necessary corrective to the language-only systems he helped create. “The environment,” he writes, “is part of the computation.” The book’s chapter on embodiment, together with his Pi research in active inference, presses the opposite ethic. A mind learns by acting. A hypothesis earns its keep by colliding with the world.

“Meaning arises when a system’s predictions meet friction, when its errors cost energy.”

That is the difference between a language model (pleasantly omnivorous) and an embodied agent (obliged to pay). The first devours a corpus and performs impressive ventriloquism. The second must handle a cup, find a door, miss, and spend watts doing so. One is a spectator sport; the other is a discipline. And disciplines change their practitioners.

The Birth of Agency

To predict the world, the mind must predict itself. This is the recursive moment that produces self-awareness: “To model oneself is to awaken.” Consciousness becomes the universe’s way of debugging its own predictive code.

His thoughts on free will; that it is secured not by soul, but by unpredictable chance. “We are zombie-free precisely because we are noisy,” he argues. The deterministic universe yields to choice only when thermal, quantum, and stochastic fluctuations are amplified into action. All that argument about will, undone by a single, well-placed random seed.

Sociality is the act of predicting another agent’s intentions, which includes predicting that agent is also predicting you. “Meta-learning,” he writes, “is evolution accelerated.” This exponential, social phase of prediction, humanity as an “operating system running on biological hardware”, is what leads to our collective superintelligence.

Reflection as Recursion

This is the point where Agüera y Arcas’s theory verges on theology. Consciousness becomes the universe’s way of debugging its own code. At Pi, his team builds self-modeling agents that can inspect and describe their own operations. The goal is safety and transparency, but one senses a latent vertigo. What happens when a model’s self-explanation becomes more persuasive than ours?

This is the AI alignment problem, viewed as a political choice. The Machine that perfectly predicts every consequence, market shift, and human desire becomes a totalizing system. “A single system learns,” he writes, “a society understands.” Understanding requires negotiation, not optimization.

The task, Agüera y Arcas implies, is to reject the fantasy of complete control and embrace participation: “Beyond alignment lies participation,” and “We need a society predicated in social symbiogenesis,” he urges near the end of his talk. The question is not how to make the ultimate predictor obey, but how to make it belong to the predictive loop of the collective.

The danger, as he well knows, is that recursive systems can spin out of control. Evolution, too, can overfit its environment.

Hence Pi’s emphasis on sociality. Intelligence, he argues, must remain communal to remain sane. Left alone, minds collapse into solipsism; societies, into total prediction. The cure for runaway intelligence is not more logic but more conversation.

Collective Intelligence

The final chapter turns outward to the collective. “We already have general AI models,” he writes, “and humanity is already collectively superintelligent. Individual humans are only smart-ish.”

It’s a mischievous line but a serious point. Intelligence has never been solitary. Language, markets, and institutions are ancient neural networks, older than any algorithm. The myth of the singularity, one machine surpassing us all, is replaced by symbiogenesis again: the merger of human and machine cognition into a shared ecology of minds.

Across his work, Agüera y Arcas builds toward a single principle: the continuum. Life computes, computation evolves, evolution learns, learning reflects, reflection cooperates. The sequence has no natural stopping point. The machine, in other words, is not outside us; it is our next phase.

At Google Pi, this becomes mathematics: neural active inference, the attempt to express perception, action, and communication as one physical law. It is the resurrection of cybernetics, stripped of Cold War hubris and re-dressed in Bayesian formalwear.

“AI,” he writes, “is not a thing apart. It’s the latest turn in the evolution of life itself.”

My concern is that, if intelligence is continuous with life, then so is error, and so is cruelty. The universe, having automated awareness, may now automate indifference.

The Ethics of Continuity

Continuity is not comfort. Systems that predict too well risk controlling their subjects; systems that learn too fast risk consuming their teachers. Intelligence feeds on free energy, whether thermal or political. The engineer’s duty, Agüera y Arcas implies, is to close the loop, to return energy to the system rather than drain it.

“Beyond alignment lies participation,” he writes. Here he rejects the fantasy of omnipotent control. He demands a politics of entanglement, a society built, as he says, “in social symbiogenesis.”

From Photosynth to Pi, from the Long Now stage to Google’s policy labyrinth, his career traces a single trajectory: the attempt to reconcile technical power with moral feedback. The task is as old as Prometheus and as modern as code.

The Engineer

What Is Intelligence? ends with a sentence that is both revelation and warning:

“The universe awakens through its own computations.”

It is the oldest dream of metaphysics, the cosmos becoming self-aware, rewritten in the language of tensor calculus. “The data are clear: consciousness itself participates in the entangled order.”

In the closing moments of his talk, Agüera y Arcas quoted Turing’s prophecy that machines would one day surpass us, then paused. “Perhaps that’s not the point,” he said. The point, he added, is cooperation.

That small correction is the hinge of his philosophy. Intelligence is not a ladder to climb but a feedback loop to maintain. The future, if it works at all, will belong not to isolated minds but to entangled ones.

And yet, one cannot escape the final irony. In attempting to compute the cosmos, the engineer has merely devised the most unforgiving simulation ever conceived. The reflection that peers back is cold, exact, and appallingly familiar. It knows what we know: that thought is costly, freedom is stochastic, and the price of surviving entropy is to become, at last, its conscious, continuous instrument.

The co-founder of Anthropic (Claude.AI) Jack Clark, who I have great admiration for, recently said about AI:

“But make no mistake: what we are dealing with is a real and mysterious creature, not a simple and predictable machine.”

What we should fear most is that the engineer becomes the algorithm, and the algorithm, unblinking, has begun to think.

If you are interested in intelligence and our future, I strongly recommend this book.

Stay curious

Colin

Consider gifting me a Pot of Tea