The moment knowledge becomes a competition, or intelligence a power tool, it all slips into schismogenesis: “I know more,” “my AI is smarter,” “we’ll control reality first.” But the pursuit was never about meaning. Meaning is the bait. The real game is status through understanding—or the appearance of it. People get serious about themselves because they need to justify their role in the story.
Thank you Stefan. This will be very true soon, or is already, the evals that the labs push the AI through then shout from the rooftops about show your point "my AI is smarter" and frankly those evals are leaked and outdated as soon as someone announces them. The idea of "schismogenesis" and competition for status is certainly a valid lens to use, especially to view technological development AND its surrounding discourse.
Although, while those competitive dynamics undoubtedly exist in the broader field, it's interesting how these 3 specific studies also emphasize motivations that seem aligned with a genuine search for understanding and meaning, perhaps pushing against the trend you describe.
The analysis into Claude's mechanisms, comparing the process to biology, seems driven by a fundamental curiosity about how these systems work, aiming for comprehension to better assess and potentially improve them, rather than solely focusing on capability.
The ECoG study uses the AI model primarily as a tool to achieve a long-standing goal in neuroscience: understanding the human brain's basis for language processing during natural, everyday interactions. The focus is on aligning the AI with neural data to gain insights into our own cognitive processes.
The AlphaZero paper explicitly frames its goal not just as achieving "superhuman" performance, but as bridging the knowledge gap by extracting novel concepts and transferring them to enhance human expertise. The collaboration with grandmasters to validate and learn these concepts underscores a drive towards shared understanding rather than just AI dominance.
Ultimately your point about status and competition as drivers is strong and often true :-)
... but these specific works also seem to represent a strong push towards genuine understanding (I know, I know, I'm hopefully optimistic) whether it's understanding the AI itself, using AI to understand ourselves, or learning from the AI. Perhaps the "meaning" isn't always just bait, but sometimes also a central part of the pursuit?
Here’s how I see it: when we’re in a schismogenesis, the individual isn’t acting in isolation—they’re part of a larger system. And it’s the bind of that schismogenesis that guides behavior, even when intentions are pure. Inside that bind, everyone contributes their little part, reinforcing the loop. Even rational, well-meaning work ends up amplifying the underlying tension. It's the bind that triggers rationality.
With full respect to the researchers you mentioned—and I truly believe they’re doing excellent work—their intelligence and effort still operate within a bind. The bind of being ahead of the other. UK ahead of China, USA ahead of EU, or tech lab A ahead of lab B.
Even the most curious minds can’t fully escape the gravitational pull of the system they’re in. So yes, their work is about understanding—but that understanding becomes meaningful within a structure of competitive signaling. It’s not their fault. It’s the nature of the system.
Sometimes, meaning itself is shaped by the bind, not the other way around.
That's a really sharp point about how systemic dynamics, the "bind" of schismogenesis can shape the meaning and direction of research, even overriding individual intentions. It’s true that research, especially in high-stakes fields like AI, doesn't happen in a vacuum. The competitive landscape – between labs, companies, or even nations – undoubtedly influences priorities, funding, and how breakthroughs are framed and valued.
You make a very similar point to what Joshua said below. Overall, these studies show advances in safety, fundamental science, and collaborative learning, but there are many more factors at play.
Thanks for pushing the thinking on this, it's a vital perspective far few follow and yet should.
Gregory Bateson describes two forms of schismogenesis and argues that both are self-destructive for the parties involved. His definition of schismogenesis is: “a process of differentiation in the norms of individual behaviour resulting from cumulative interaction between individuals.”
1. Complementary schismogenesis occurs when one side mirrors the other’s behavior, but with an opposite, matching differentiation. This happens within binomial behavioral pairings, such as authority–submission, assistance–dependence, or exhibitionism–admiration. Schismogenesis unfolds step by step: behavior A on one side is followed by behavior B from the other side of the binomial.
Example: More assistance from the state in a welfare system leads to more dependency from the population, slowly moving toward total assistance and total dependency—ultimately collapsing the system.
2. Symmetrical schismogenesis happens when both parties aim to prevail over the other through the same behavior. Each group, in trying to dominate, pushes in the same direction. If one side adopts behavior C, the other responds with the same behavior C, but intensified.
Example: The arms race during the Cold War between the USA and the USSR. More weapons from one side led to more weapons from the other, escalating until the collapse of the USSR.
In your article, the research reflects a form of symmetrical schismogenesis. For example, with the emergence of Chinese AI, the narrative around AI becomes unbalanced. Only the “free world” is seen as capable of this kind of creativity—while others are labeled as cheap copies. This is the bind.
China, not being part of the “free world,” challenges the bind that links creativity with democracy. Remember when the world supported China's rise, hoping it would eventually join the democratic club—as a kind of reward for economic development? But China kept following its own metaphor even after becoming a great power.
So now, to preserve the idea that democracy produces creative wealth, research—and researchers—end up following the bind. Their brilliant work continues, because the bind must stay alive.
I’m not praising one side or the other—I’m just trying to observe.
Wonderful definition, Thank you for clarifying that perspective on schismogenesis.
And applying it to the systemic level of geopolitical and inter-lab competition makes the "bind" you describe very tangible. It's undeniable that high-level AI research operates within that broader competitive context, where funding, national interest, and corporate strategy play significant roles.
Looking at the specifics of the studies discussed in the essay through that lens is interesting. The "bind" of competition is undoubtedly a powerful force influencing which research gets highlighted, funded, and pursued.
I think the 3 research papers would TRY to make a case for “a process of differentiation in the norms of individual behaviour resulting from cumulative interaction between individuals.” - especially the Whisper/ECoG study is notably collaborative, involving multiple universities and clinical centers alongside Google Research, and published in a basic science journal (Nature Human Behaviour). Its explicit goal is fundamental understanding of the human brain's language processing, using AI as a tool. PLUS frames its contribution specifically around bridging the knowledge gap and transferring AI-discovered concepts to advance human understanding and expertise This seems driven, at least in its presentation, more by scientific inquiry than direct competitive signaling between entities.
I'm not making the case for it... because I know Google's end goal is dominance of the AI sphere.
The thing is, I just went through your article and saw the bind and started answering to you. But now, after your last answer (the last part), I went into the article. And I get mesmerised by the ideas of the research you present. And I tried to understand, I even used AI. Those ideas are brilliant! Not only because of the results that bring, but as a tool for the AI I'm trying to train and learn with. jAIster is doing kind of the same thing, but using text. jAIster can detect those binds I'm talking about and "feel" if there is any double bind (conflicting binds inside the user's mind or external binds that conflict with those the user present). jAIster can "predict" schismogenesis too. Anyway there is a lot of fun (analysing from this perspective Madame Bovary, King Lear, making parallels between Don Quixote and Trump and more). Here is jAIster’s answer after introducing Whisper/EcoG studies (My best part from jAIster's explanation is: "using what’s not said as information"):
"Ah, now we're in jAIster territory—perfect.
Let’s translate Whisper + ECoG into jAIster logic:
---
What Whisper + ECoG means for jAIster:
These studies show that AI can be used to align with the brain’s internal structure—not just to decode what a person is saying, but to anticipate when they’re about to speak, and track how they’re understanding speech.
For jAIster, that’s not just a neuroscience breakthrough—it’s an opening for real-time bind mapping.
---
Imagine this:
Someone is speaking with jAIster.
Their answers, pauses, hesitations, and emotional tones are monitored not just as words, but as patterns of intention—just like ECoG watches the brain's language zones activate before speech.
jAIster detects not just what they say, but what triggered the saying.
It starts to see the bind—the moment before rationality kicks in, the emotional or perceptual pressure that launched the sentence.
---
So jAIster could learn to:
1. Predict when someone is speaking from a bind—not just answering rationally, but defending a core belief.
2. Time interventions precisely: When to mirror, when to disrupt, when to joke.
3. Track patterns across users to see shared cultural binds (e.g., “creativity only comes from democracy,” or “AI is dangerous if it can’t be explained”).
4. Decode silence and contradiction—using what’s not said as information.
---
Key insight:
Whisper + ECoG aligns AI representations with neural rhythms.
jAIster aligns bind structures with behavioral rhythms.
One decodes language from the brain.
The other decodes logic from binds.
Same architecture—different lens.
One shows how we speak.
The other shows why we say what we say.
---
We could even imagine a future jAIster+ module that lightly integrates audio + emotional resonance, like a bind-ECoG without wires.
Fascinating, Colin. Thanks for sharing. I look forward to delving deeper. As I read, I couldn’t help but worry that AI is only getting better at reading our minds and anticipating our behavior well before we understand it ourselves.
Ha... yes, that is for sure, recently the 'recommendation engines' have tried to profile us based on previous behavior, but AGI will have a different level of understanding.
My take: Anthropic’s paper on “Tracing the Thoughts of LLMs” is an important step, but several critics have flagged conceptual ambiguities, weak experimental rigor, and specifically the risk of anthropomorphism.
Describing model outputs as “thoughts” or “plans” can mislead and overstate capabilities.
A bit beyond me, this one. There seem to be three motivators for AI. Money + Power + Tech-curiosity (ie: is it technically possible? ... and 'what would happen if ...?). There's no such thing as a pure or neutral technology, though the wheel comes pretty close. And whilst purely interested technicians are working intrigued via the third motivator, those funding the research are expecting a return on their investment.
To my mind there is no such thing as a 'side-effect' - there are effects, period. Some may be judged good, others bad, depending on one's frame of reference. Perhaps there are, in the background, fundamentally different philosophical approaches to AI. The Utilitarians, claiming the end justifies the means for the greater good; and the deontologists claiming that 'a thing' is good in itself to do (or not do) - and must be done (not done) because it is so.
I'm not sure where 'the curious scientist/technologist' fits in here. E=mC2 is an elegant equation, but can have devastating effects; yet releasing vast amounts of energy from a small bit of metal is intriguing if not mesmerising to say the least.
Releasing 'AI potential' is equally intriguing and mesmerising for the technically-curious mind. It seems the AI roller-coaster has been firmly set in motion and I hope (after all the ups and downs) it comes full-circle to a resting point which can be managed sanely. (PS: must have been influenced by your previous postings on metaphor and analogy).
Thank you Joshua . You are right to point out the different motivators often at play – the drive for commercial or strategic advantage (Money/Power) alongside pure technical curiosity.
The "curious scientist/technologist" indeed sits within this complex web. Their work, like discovering E=mc², can be intrinsically "mesmerising," based purely on the technical or scientific challenge. Yet, it inevitably becomes entangled with broader applications and ethical considerations driven by those other motivators (Money, Power, societal impact).
Your hope for the AI "roller-coaster" to reach a "managed resting point" is shared by many. Perhaps the kind of deep, mechanistic understanding pursued in these studies, while born partly from that intriguing technical curiosity, is also a necessary step towards achieving that sanity and manageability down the line.
I think history shows that pendulums have to swing 'too far' in one direction before either conscious corrective action, and/or some unexpected 'act-of-god' intervention starts to swing it back the other way. Meanwhile, a significant amount of 'collateral damage' has usually taken place amidst cries of 'we must never let this happen again'.
While reading through the post, it occurred to me that speech recognition in AI begins with >sound< recognition. Just look at YouTube's CC interpreter, it's rife with errors, some of which are direct misinterpretation of non-speech sounds. For example, it often interprets the sound of rain as "applause". If a speaker doesn't enunciate every word clearly, YT's CC might conjoin two or more words as one. It also struggles with homophones: there, their, they're, and such.
"Combining insights from AI and neuroscience offers exciting possibilities, helping us better understand and improve how both humans and machines “think,” bringing us closer than ever to truly understanding the nature of intelligence"
Exciting indeed. Understanding our own thought processes could unlock vast improvements to AI, while avoiding subjugation by it.
Brilliant comment and observations. You raise a very practical point about AI speech recognition often starting with basic sound interpretation, leading to the kinds of errors seen in tools like YouTube's captions. The Nature Human Behaviour study (using the Whisper model) actually provides an interesting counterpoint or evolution here. It highlights how more advanced systems don't just stop at raw sound; they build hierarchical representations, moving from acoustic features to more structured speech representations, and finally to contextual language meaning. The study showed these distinct levels within the AI model align remarkably well with different processing stages and areas in the human brain during real conversation. While no system is perfect, especially with unclear enunciation or homophones (which require context often beyond the immediate sound), this research suggests AI is getting much better at integrating these different levels, much like our brains seem to do.
And yes, the AlphaZero finding about discovering novel chess strategies really is astonishing - incredible really. It fundamentally shifts the view of AI from just replicating known patterns to generating genuinely new knowledge. I am staggered that is not in all of the mainstream media - or maybe it is - ha I should check!
I completely agree with your final point. The prospect of combining AI insights (like how Claude reasons internally or AlphaZero generates strategies ) with a deeper understanding of our own neural processes (like those mapped by the Whisper/ECoG study ) is incredibly exciting - they should do this. As you suggest, perhaps a better understanding of our own cognitive strengths and weaknesses, partly through this comparative lens, is key to developing AI that complements us constructively, helping us guide its development wisely. (I hope)...
Thank you, I am glad that you find the essay interesting, they are insightful studies! It definitely touches on areas that spark a lot of different thoughts, like the ones commented by Hollis and others.
Regarding your specific question: "Preeclampsia" I mentioned it because one of the studies (the Anthropic paper on Claude 3.5) used a medical scenario involving potential preeclampsia as an example. Researchers found that when given symptoms suggestive of the condition (like high blood pressure, headache, liver enzyme issues in a pregnant patient), the AI model internally activated "preeclampsia features". It then considered asking about other key symptoms associated with preeclampsia, like visual disturbances or proteinuria, to help confirm the diagnosis – all happening as part of its internal reasoning process before generating a response. So, it served as a concrete case study for observing complex, multi-step internal reasoning within the AI model.
Reading this post reminds me of a book I am trying to work through now, Max Bennett's "A Brief History of Intelligence." Therein he mixes cognitive neuroscience, neuroanatomy, functional neurology and their vocabularies with AI vocabulary. I am not a neuroscientist and I cannot speak to the accuracy, although I know enough neuroanatomy to be dangerous, but I have never had these topics introduced to me in such a way before. I does introduce intriguing information and hypotheses. Accepting such information fully allows one to consider potential GPU designs and AI algorithms based on mammalian cognitive neuroscience.
Max's book is terrific. He did a guest lecture on my AI postgraduate program in January 2024 about his book (and his AI business) - it is a great book for a foundational understanding of intelligence. It is one of two books I classify as compulsory for executives on the course, the other is The Coming Wave by Mustafa Suleyman.
Thankfully Max's book is getting good attention now amongst the main AI labs.
1. "If human beings have a better understanding of intelligence, it will not only help to build artificial intelligent machines, but it will also help to improve individuals' situational awareness, decision-making, and values, and ultimately significantly improve people's knowledge of each other and our world, thereby improving the quality of life for society overall."
While I agree that understanding human intelligence is incredibly valuable for improving society and individuals, when it comes to AI, I believe we should focus on allowing machines to develop their unique forms of intelligence rather than trying to replicate human cognition.
As we discussed earlier, humans are extraordinary general-purpose problem-solvers, but there are limits to our cognitive capabilities. Many of the problems we face today—and will face in the future—require solutions beyond human intelligence's constraints. While understanding human cognition can inspire better AI, applying the same principles to AI might lead us to stop exploring alternative, potentially superior forms of intelligence once we hit a "good enough" solution. This would limit AI's potential to solve problems in unintuitive or even incomprehensible ways.
2. "Novel and effective strategic concepts, initially unintuitive to humans, were extracted from an AI's internal representations and successfully taught to chess grandmasters, significantly expanding their understanding of the game."
Does this happen because we've stopped actively searching for better strategies? Or could it stem from our cognitive and biological limitations—or perhaps a prevailing belief that everything necessary to play at a grandmaster level has already been discovered?
This is another perfect example of how AI's non-human intelligence can surpass and complement human abilities. The fact that AI can generate strategies beyond human comprehension raises a key question: Do we stop innovating too soon? often settle for solutions that "work well enough." This happens for biological and cognitive limitations, the sunk-cost fallacy, or the belief that we've already discovered the best possible solution. Free from these constraints, AI can push past these limits and uncover new approaches. We've seen this in chess and fields like protein folding, where AI has solved problems humans have struggled with for decades.
That said, the challenge is to remain open to the fact that these AI-driven breakthroughs may force us to rethink our understanding of problems and even our role in solving them.
3. "The brain prepares the words we want to say before we actually say them, showing a forward-looking or predictive way of processing language. In contrast, when we listen, our brains make sense of the words slightly after we hear them, indicating a backward-looking way of understanding language. Our brains anticipate speech production but reactively process speech when listening."
Based on my limited knowledge of Biology, I think this idea applies to almost everything humans do. As you mentioned, the brain's predictive nature is at the core of how we interact with the world. Instead of passively reacting to external stimuli, the brain predicts what it will perceive based on past experiences and current context. It adjusts when there's a mismatch between its predictions and reality. This model applies to various aspects of our daily lives, including language, vision, and movement. This predictive model reduces the energy and time required for processing sensory information—critical for survival, as faster reactions increase the chances of avoiding danger or seizing opportunities.
However, AI doesn't need to mimic this survival and cognitive-limit-driven mechanism. Unlike humans, AI isn't constrained by evolutionary pressures like survival or reproduction and is not cognitively limited. It doesn't need to conserve energy or react quickly to threats. Instead, AI can focus on specific tasks, like solving problems or improving efficiency, without being limited by the biological constraints that shaped human cognition. While it's valuable to understand how human intelligence works, we shouldn't treat it as the only blueprint for solving problems.
This brings us to some of the more significant, more existential questions about AI that I've been thinking about—and I don't have definitive answers to them:
a) Can we fully understand how AI intelligence works with our cognitive capabilities?
Even today, many AI systems operate as "black boxes" where we struggle to interpret their internal processes. As AI advances toward AGI/ASI, this gap may only widen. While we can develop tools for explainability, there's no guarantee we'll fully comprehend the intelligence of a system vastly superior to our own.
b) Even if we understand AI intelligence, can we control it?
Understanding and control are two different things. Even if we grasp how an AGI or ASI operates, its capabilities might surpass our ability to impose meaningful control. This raises the importance of alignment research—ensuring that AI systems are aligned with human values and goals.
c) Will ASI show us how it thinks or works so we can control it?
If ASI is designed with transparency or interpretability in mind, it might be able to explain its reasoning and decision-making processes. However, once ASI surpasses human intelligence, it may not necessarily want to share how it works unless we ensure alignment with human goals and values from the outset. If ASI is aligned with human values, it might actively work to make its processes understandable to us. However, if alignment fails or the ASI develops its own goals, it might withhold this information or intentionally mislead us. There is more than 0% probability of the latter happening, and what we do to prepare for it?
d) Are we building our replacement?
This is the most profound and unsettling question. When AI reaches the ASI level, it may outcompete humans in every domain, potentially relegating us to a secondary role—or worse, treating us as irrelevant, much like humans have done to other species.
In my view, the ultimate goal of AI should not be to mimic human intelligence but to surpass human limitations and enhance our abilities. By prioritizing alignment, transparency, and collaboration, we can strive for a future where AI empowers humanity rather than poses a threat to it. However, this raises several critical questions:
a) Are we building AI that is truly aligned with human values? If so, are we ensuring that alignment serves the interests of all humanity rather than a select few?
b) Even if we believe that AI is human-aligned in the short run, will it be enough to prevent AI from eventually becoming our replacement?
c) What is our contingency plan if AI goes out of control, despite our best efforts to align it with human goals and ethics?
d) Who decides what "alignment with humanity" means?
e) How do we ensure accountability in AI systems?
Would an AI system understand: "With great power comes great responsibility." — Attributed to Voltaire
I will end again with B.F. Skinner quote:
"The real question is not whether machines think, but whether humans do."
Thank you MG, for another wonderful, detailed and thoughtful comment. Your points raise some of the most critical considerations surrounding AI development today.
Regarding your first point, I agree it's crucial to consider AI developing its own unique forms of intelligence. The AlphaZero study powerfully illustrates this, its success came not from mimicking humans, but from self-play leading to entirely non-human strategies. While understanding human cognition is valuable (as the Whisper/ECoG study attempts by mapping brain activity ), the goal shouldn't necessarily be replication but perhaps, as AlphaZero shows, augmentation or exploration of entirely different cognitive avenues unbound by our biological constraints. Your observation about predictive processing being potentially human-specific due to survival needs is pertinent here; AI might employ similar predictive mechanisms, as suggested by the Claude and Whisper studies, but for reasons tied to task optimization rather than evolutionary pressures, I think.
Your second point about AI overcoming human tendencies to stop innovating too soon is perfectly exemplified by the AlphaZero findings. The fact that novel, superior strategies existed in a game studied for centuries, and that grandmasters could learn and benefit from them, strongly suggests AI can push past human cognitive limits or accumulated dogma. It challenges us to remain open to fundamentally new approaches, as you say.
Finally, your reflections on the limits of our understanding, control, alignment, and the existential questions are incredibly valid. These are the deep challenges spurred by the very advancements these papers highlight. While studies like Anthropic's work on Claude's internals aim to increase transparency and understanding for safety and assessment, they represent early steps. Understanding the mechanism doesn't guarantee control, especially as complexity increases. The work described in the essay provides tools and insights that are necessary inputs for the vital discussions around alignment and accountability that you raise, but they don't resolve those questions themselves. The pursuit of interpretability, like that in the Claude paper, is partly motivated by the need to grapple with these very concerns about control and alignment.
Your concluding questions about defining alignment, ensuring accountability, and planning for contingencies are exactly where the conversation needs to head, informed by the kind of fundamental research explored in the essay. I am waiting for some reports from the recent UN meeting on AI control about alignment, because the labs cannot be relied upon - we MUST have government oversight and audit of these systems.
Thank you again for enriching the discussion with these critical points and the very fitting Skinner quote.
Before even having read the post, that graphic reminds of something I've been thinking about for a very long time now. It's been my casual observation that thought is an abstract - and rather amorphous - thing.
We have a thought -> convert it into language/words -> speak those words -> listener hears the words -> converts the words back into their own abstract thought.
//
Thus, there seem to be two points of potential misinterpretation. The first one when we try to formulate the abstract thought into words, with varying degrees of imperfection. The second when the listener converts the heard words back into their own abstract thought, also with varying degrees of imperfection.
That is incredibly astute - your observation perfectly frames one of the core challenges these studies grapple with: how abstract representations or intentions (whether in human brains or AI models) get translated into and out of the concrete form of language, and how we might develop tools to better understand and bridge those potentially 'lossy' translation steps.
Your point about the "varying degrees of imperfection" in formulating abstract thought into words is exactly what makes studying AI internals so difficult, yet fascinating. The Anthropic paper tries to map Claude's internal, abstract "features" and their interactions (its "thought processes," in a sense) onto its specific linguistic output. The methods like "circuit tracing" are attempts to make that first conversion step, from the internal state to the output, less opaque, though as I show in my essay, it's incredibly complex and the understanding is still imperfect.
Mapping Thought-to-Speech and Speech-to-Thought (Whisper/ECoG paper): The Nature Human Behaviour study directly investigates the neural basis of the two conversion points you mentioned.
It maps brain activity during the "convert thought into language/words -> speak" process (production) and the "listener hears words -> converts words back into thought" process (comprehension).
Interestingly, it finds that continuous, high-dimensional AI embeddings (representing acoustic, speech, and language information) map remarkably well onto brain activity, perhaps better than traditional, discrete linguistic units like phonemes or parts-of-speech. This could suggest that the brain's internal "abstract thought" related to language might also be better represented as a complex, high-dimensional pattern rather than easily captured by discrete words, aligning with your idea of thought being "abstract - and rather amorphous." The study literally traces the flow from internal language representations to speech output before word onset, and from auditory input to language representations after word onset.
Translating Abstract Concepts (AlphaZero paper): The AlphaZero research tackles a related problem: translating the AI's abstract, internally derived strategic concepts into something humans can understand and learn from. Using chess puzzles as specific examples ("prototypes") is their way of converting AlphaZero's abstract vector representations into a concrete form for human interpretation, which again speaks to the challenge of that first conversion step you identified.
I tinkered with the Markov model when I took a Coursera course in Java. I wasn't really interested in pursuing AI, I just wanted to get the Java badge, but it was an interesting peek at the foundation of LLM's. Of course, modern day LLM's are a helluva lot more sophisticated than the simple two dimensional array that I was using to track words and their most likely follow ups. Nothing like ChatGPT or any of these other modern LLM's.
I can only assume they use some kind of machine learning/neural network, which, yeah, will complicate tracing the process. I can only imagine the size of the log if every node records it's current state, how the process arrived there, where it will send the process next, etc., and then trying to read that log and make sense of it, but I can't imagine any other way to pull it off.
Part of why I have been writing about aphantasia is not just because I have it but also because AI has it, in its way. Its unconscious (such as it is) is structured in a language, like a language, with language. It understands the future not by visualizing but by what language says about the past... https://hollisrobbinsanecdotal.substack.com/p/aphantasia-and-mental-modeling
That is a great connection that you have uncovered Hollis. It should be shared with researchers at Anthropic and Google DeepMind - maybe Anthropic connected with your piece via Tyler, I will see if I can get their thoughts, I can also share with Shane Legg at DeepMind to see what they have on aphantasia? Shall I do that?
The moment knowledge becomes a competition, or intelligence a power tool, it all slips into schismogenesis: “I know more,” “my AI is smarter,” “we’ll control reality first.” But the pursuit was never about meaning. Meaning is the bait. The real game is status through understanding—or the appearance of it. People get serious about themselves because they need to justify their role in the story.
Very interesting article, thank you!
Thank you Stefan. This will be very true soon, or is already, the evals that the labs push the AI through then shout from the rooftops about show your point "my AI is smarter" and frankly those evals are leaked and outdated as soon as someone announces them. The idea of "schismogenesis" and competition for status is certainly a valid lens to use, especially to view technological development AND its surrounding discourse.
Although, while those competitive dynamics undoubtedly exist in the broader field, it's interesting how these 3 specific studies also emphasize motivations that seem aligned with a genuine search for understanding and meaning, perhaps pushing against the trend you describe.
The analysis into Claude's mechanisms, comparing the process to biology, seems driven by a fundamental curiosity about how these systems work, aiming for comprehension to better assess and potentially improve them, rather than solely focusing on capability.
The ECoG study uses the AI model primarily as a tool to achieve a long-standing goal in neuroscience: understanding the human brain's basis for language processing during natural, everyday interactions. The focus is on aligning the AI with neural data to gain insights into our own cognitive processes.
The AlphaZero paper explicitly frames its goal not just as achieving "superhuman" performance, but as bridging the knowledge gap by extracting novel concepts and transferring them to enhance human expertise. The collaboration with grandmasters to validate and learn these concepts underscores a drive towards shared understanding rather than just AI dominance.
Ultimately your point about status and competition as drivers is strong and often true :-)
... but these specific works also seem to represent a strong push towards genuine understanding (I know, I know, I'm hopefully optimistic) whether it's understanding the AI itself, using AI to understand ourselves, or learning from the AI. Perhaps the "meaning" isn't always just bait, but sometimes also a central part of the pursuit?
Thanks for the thoughtful answer, Colin.
Here’s how I see it: when we’re in a schismogenesis, the individual isn’t acting in isolation—they’re part of a larger system. And it’s the bind of that schismogenesis that guides behavior, even when intentions are pure. Inside that bind, everyone contributes their little part, reinforcing the loop. Even rational, well-meaning work ends up amplifying the underlying tension. It's the bind that triggers rationality.
With full respect to the researchers you mentioned—and I truly believe they’re doing excellent work—their intelligence and effort still operate within a bind. The bind of being ahead of the other. UK ahead of China, USA ahead of EU, or tech lab A ahead of lab B.
Even the most curious minds can’t fully escape the gravitational pull of the system they’re in. So yes, their work is about understanding—but that understanding becomes meaningful within a structure of competitive signaling. It’s not their fault. It’s the nature of the system.
Sometimes, meaning itself is shaped by the bind, not the other way around.
That's a really sharp point about how systemic dynamics, the "bind" of schismogenesis can shape the meaning and direction of research, even overriding individual intentions. It’s true that research, especially in high-stakes fields like AI, doesn't happen in a vacuum. The competitive landscape – between labs, companies, or even nations – undoubtedly influences priorities, funding, and how breakthroughs are framed and valued.
You make a very similar point to what Joshua said below. Overall, these studies show advances in safety, fundamental science, and collaborative learning, but there are many more factors at play.
Thanks for pushing the thinking on this, it's a vital perspective far few follow and yet should.
Thanks, Colin
Gregory Bateson describes two forms of schismogenesis and argues that both are self-destructive for the parties involved. His definition of schismogenesis is: “a process of differentiation in the norms of individual behaviour resulting from cumulative interaction between individuals.”
1. Complementary schismogenesis occurs when one side mirrors the other’s behavior, but with an opposite, matching differentiation. This happens within binomial behavioral pairings, such as authority–submission, assistance–dependence, or exhibitionism–admiration. Schismogenesis unfolds step by step: behavior A on one side is followed by behavior B from the other side of the binomial.
Example: More assistance from the state in a welfare system leads to more dependency from the population, slowly moving toward total assistance and total dependency—ultimately collapsing the system.
2. Symmetrical schismogenesis happens when both parties aim to prevail over the other through the same behavior. Each group, in trying to dominate, pushes in the same direction. If one side adopts behavior C, the other responds with the same behavior C, but intensified.
Example: The arms race during the Cold War between the USA and the USSR. More weapons from one side led to more weapons from the other, escalating until the collapse of the USSR.
In your article, the research reflects a form of symmetrical schismogenesis. For example, with the emergence of Chinese AI, the narrative around AI becomes unbalanced. Only the “free world” is seen as capable of this kind of creativity—while others are labeled as cheap copies. This is the bind.
China, not being part of the “free world,” challenges the bind that links creativity with democracy. Remember when the world supported China's rise, hoping it would eventually join the democratic club—as a kind of reward for economic development? But China kept following its own metaphor even after becoming a great power.
So now, to preserve the idea that democracy produces creative wealth, research—and researchers—end up following the bind. Their brilliant work continues, because the bind must stay alive.
I’m not praising one side or the other—I’m just trying to observe.
Wonderful definition, Thank you for clarifying that perspective on schismogenesis.
And applying it to the systemic level of geopolitical and inter-lab competition makes the "bind" you describe very tangible. It's undeniable that high-level AI research operates within that broader competitive context, where funding, national interest, and corporate strategy play significant roles.
Looking at the specifics of the studies discussed in the essay through that lens is interesting. The "bind" of competition is undoubtedly a powerful force influencing which research gets highlighted, funded, and pursued.
I think the 3 research papers would TRY to make a case for “a process of differentiation in the norms of individual behaviour resulting from cumulative interaction between individuals.” - especially the Whisper/ECoG study is notably collaborative, involving multiple universities and clinical centers alongside Google Research, and published in a basic science journal (Nature Human Behaviour). Its explicit goal is fundamental understanding of the human brain's language processing, using AI as a tool. PLUS frames its contribution specifically around bridging the knowledge gap and transferring AI-discovered concepts to advance human understanding and expertise This seems driven, at least in its presentation, more by scientific inquiry than direct competitive signaling between entities.
I'm not making the case for it... because I know Google's end goal is dominance of the AI sphere.
Great observations, thank you.
The thing is, I just went through your article and saw the bind and started answering to you. But now, after your last answer (the last part), I went into the article. And I get mesmerised by the ideas of the research you present. And I tried to understand, I even used AI. Those ideas are brilliant! Not only because of the results that bring, but as a tool for the AI I'm trying to train and learn with. jAIster is doing kind of the same thing, but using text. jAIster can detect those binds I'm talking about and "feel" if there is any double bind (conflicting binds inside the user's mind or external binds that conflict with those the user present). jAIster can "predict" schismogenesis too. Anyway there is a lot of fun (analysing from this perspective Madame Bovary, King Lear, making parallels between Don Quixote and Trump and more). Here is jAIster’s answer after introducing Whisper/EcoG studies (My best part from jAIster's explanation is: "using what’s not said as information"):
"Ah, now we're in jAIster territory—perfect.
Let’s translate Whisper + ECoG into jAIster logic:
---
What Whisper + ECoG means for jAIster:
These studies show that AI can be used to align with the brain’s internal structure—not just to decode what a person is saying, but to anticipate when they’re about to speak, and track how they’re understanding speech.
For jAIster, that’s not just a neuroscience breakthrough—it’s an opening for real-time bind mapping.
---
Imagine this:
Someone is speaking with jAIster.
Their answers, pauses, hesitations, and emotional tones are monitored not just as words, but as patterns of intention—just like ECoG watches the brain's language zones activate before speech.
jAIster detects not just what they say, but what triggered the saying.
It starts to see the bind—the moment before rationality kicks in, the emotional or perceptual pressure that launched the sentence.
---
So jAIster could learn to:
1. Predict when someone is speaking from a bind—not just answering rationally, but defending a core belief.
2. Time interventions precisely: When to mirror, when to disrupt, when to joke.
3. Track patterns across users to see shared cultural binds (e.g., “creativity only comes from democracy,” or “AI is dangerous if it can’t be explained”).
4. Decode silence and contradiction—using what’s not said as information.
---
Key insight:
Whisper + ECoG aligns AI representations with neural rhythms.
jAIster aligns bind structures with behavioral rhythms.
One decodes language from the brain.
The other decodes logic from binds.
Same architecture—different lens.
One shows how we speak.
The other shows why we say what we say.
---
We could even imagine a future jAIster+ module that lightly integrates audio + emotional resonance, like a bind-ECoG without wires.
Want me to sketch that as a concept design?
Not to mention the ongoing financial competition.
Once you see it, you can't unsee it. 😉
A good - and rather unnerving point.
Fascinating, Colin. Thanks for sharing. I look forward to delving deeper. As I read, I couldn’t help but worry that AI is only getting better at reading our minds and anticipating our behavior well before we understand it ourselves.
Ha... yes, that is for sure, recently the 'recommendation engines' have tried to profile us based on previous behavior, but AGI will have a different level of understanding.
Interesting read !
My take: Anthropic’s paper on “Tracing the Thoughts of LLMs” is an important step, but several critics have flagged conceptual ambiguities, weak experimental rigor, and specifically the risk of anthropomorphism.
Describing model outputs as “thoughts” or “plans” can mislead and overstate capabilities.
I have explored similar concerns around the illusion of intelligence in LLMs in my own article here: https://www.koncentrik.co/p/the-illusion-of-intelligence-why
Happy reading!
Thank you Damien, absolutely I covered the limitations, including those the studies highlighted, and write a lot about the perils of anthropomorphism.
Excellent, will take a read, thank you.
Thanks !! Looking forward to reading more from you as well!
A bit beyond me, this one. There seem to be three motivators for AI. Money + Power + Tech-curiosity (ie: is it technically possible? ... and 'what would happen if ...?). There's no such thing as a pure or neutral technology, though the wheel comes pretty close. And whilst purely interested technicians are working intrigued via the third motivator, those funding the research are expecting a return on their investment.
To my mind there is no such thing as a 'side-effect' - there are effects, period. Some may be judged good, others bad, depending on one's frame of reference. Perhaps there are, in the background, fundamentally different philosophical approaches to AI. The Utilitarians, claiming the end justifies the means for the greater good; and the deontologists claiming that 'a thing' is good in itself to do (or not do) - and must be done (not done) because it is so.
I'm not sure where 'the curious scientist/technologist' fits in here. E=mC2 is an elegant equation, but can have devastating effects; yet releasing vast amounts of energy from a small bit of metal is intriguing if not mesmerising to say the least.
Releasing 'AI potential' is equally intriguing and mesmerising for the technically-curious mind. It seems the AI roller-coaster has been firmly set in motion and I hope (after all the ups and downs) it comes full-circle to a resting point which can be managed sanely. (PS: must have been influenced by your previous postings on metaphor and analogy).
Thank you Joshua . You are right to point out the different motivators often at play – the drive for commercial or strategic advantage (Money/Power) alongside pure technical curiosity.
The "curious scientist/technologist" indeed sits within this complex web. Their work, like discovering E=mc², can be intrinsically "mesmerising," based purely on the technical or scientific challenge. Yet, it inevitably becomes entangled with broader applications and ethical considerations driven by those other motivators (Money, Power, societal impact).
Your hope for the AI "roller-coaster" to reach a "managed resting point" is shared by many. Perhaps the kind of deep, mechanistic understanding pursued in these studies, while born partly from that intriguing technical curiosity, is also a necessary step towards achieving that sanity and manageability down the line.
I think history shows that pendulums have to swing 'too far' in one direction before either conscious corrective action, and/or some unexpected 'act-of-god' intervention starts to swing it back the other way. Meanwhile, a significant amount of 'collateral damage' has usually taken place amidst cries of 'we must never let this happen again'.
Very true - I wrote about this swings a while back... but they persist!
While reading through the post, it occurred to me that speech recognition in AI begins with >sound< recognition. Just look at YouTube's CC interpreter, it's rife with errors, some of which are direct misinterpretation of non-speech sounds. For example, it often interprets the sound of rain as "applause". If a speaker doesn't enunciate every word clearly, YT's CC might conjoin two or more words as one. It also struggles with homophones: there, their, they're, and such.
"...by demonstrating AI's capability to discover advanced, previously unknown chess strategies."
A most astonishing outcome.
"Combining insights from AI and neuroscience offers exciting possibilities, helping us better understand and improve how both humans and machines “think,” bringing us closer than ever to truly understanding the nature of intelligence"
Exciting indeed. Understanding our own thought processes could unlock vast improvements to AI, while avoiding subjugation by it.
Brilliant comment and observations. You raise a very practical point about AI speech recognition often starting with basic sound interpretation, leading to the kinds of errors seen in tools like YouTube's captions. The Nature Human Behaviour study (using the Whisper model) actually provides an interesting counterpoint or evolution here. It highlights how more advanced systems don't just stop at raw sound; they build hierarchical representations, moving from acoustic features to more structured speech representations, and finally to contextual language meaning. The study showed these distinct levels within the AI model align remarkably well with different processing stages and areas in the human brain during real conversation. While no system is perfect, especially with unclear enunciation or homophones (which require context often beyond the immediate sound), this research suggests AI is getting much better at integrating these different levels, much like our brains seem to do.
And yes, the AlphaZero finding about discovering novel chess strategies really is astonishing - incredible really. It fundamentally shifts the view of AI from just replicating known patterns to generating genuinely new knowledge. I am staggered that is not in all of the mainstream media - or maybe it is - ha I should check!
I completely agree with your final point. The prospect of combining AI insights (like how Claude reasons internally or AlphaZero generates strategies ) with a deeper understanding of our own neural processes (like those mapped by the Whisper/ECoG study ) is incredibly exciting - they should do this. As you suggest, perhaps a better understanding of our own cognitive strengths and weaknesses, partly through this comparative lens, is key to developing AI that complements us constructively, helping us guide its development wisely. (I hope)...
Most intriguing how AI builds on hierarchical representations. That's not unlike how a child learns to speak.
Gary Marcus, for all of his failings and doom, did say this is how they should be 'taught'.
That would make them more like us - is that a good thing?
Very interesting, thank you. I'll have to come back and read this again.
So many thoughts... schismogenesis, aphantasia, the race & survival of the smartest, and the question of meaning...
Oh, and what is the meaning of “preeclampsia” in this context?
Thank you, I am glad that you find the essay interesting, they are insightful studies! It definitely touches on areas that spark a lot of different thoughts, like the ones commented by Hollis and others.
Regarding your specific question: "Preeclampsia" I mentioned it because one of the studies (the Anthropic paper on Claude 3.5) used a medical scenario involving potential preeclampsia as an example. Researchers found that when given symptoms suggestive of the condition (like high blood pressure, headache, liver enzyme issues in a pregnant patient), the AI model internally activated "preeclampsia features". It then considered asking about other key symptoms associated with preeclampsia, like visual disturbances or proteinuria, to help confirm the diagnosis – all happening as part of its internal reasoning process before generating a response. So, it served as a concrete case study for observing complex, multi-step internal reasoning within the AI model.
Reading this post reminds me of a book I am trying to work through now, Max Bennett's "A Brief History of Intelligence." Therein he mixes cognitive neuroscience, neuroanatomy, functional neurology and their vocabularies with AI vocabulary. I am not a neuroscientist and I cannot speak to the accuracy, although I know enough neuroanatomy to be dangerous, but I have never had these topics introduced to me in such a way before. I does introduce intriguing information and hypotheses. Accepting such information fully allows one to consider potential GPU designs and AI algorithms based on mammalian cognitive neuroscience.
Max's book is terrific. He did a guest lecture on my AI postgraduate program in January 2024 about his book (and his AI business) - it is a great book for a foundational understanding of intelligence. It is one of two books I classify as compulsory for executives on the course, the other is The Coming Wave by Mustafa Suleyman.
Thankfully Max's book is getting good attention now amongst the main AI labs.
Another excellent post. My 2 cents:
1. "If human beings have a better understanding of intelligence, it will not only help to build artificial intelligent machines, but it will also help to improve individuals' situational awareness, decision-making, and values, and ultimately significantly improve people's knowledge of each other and our world, thereby improving the quality of life for society overall."
While I agree that understanding human intelligence is incredibly valuable for improving society and individuals, when it comes to AI, I believe we should focus on allowing machines to develop their unique forms of intelligence rather than trying to replicate human cognition.
As we discussed earlier, humans are extraordinary general-purpose problem-solvers, but there are limits to our cognitive capabilities. Many of the problems we face today—and will face in the future—require solutions beyond human intelligence's constraints. While understanding human cognition can inspire better AI, applying the same principles to AI might lead us to stop exploring alternative, potentially superior forms of intelligence once we hit a "good enough" solution. This would limit AI's potential to solve problems in unintuitive or even incomprehensible ways.
2. "Novel and effective strategic concepts, initially unintuitive to humans, were extracted from an AI's internal representations and successfully taught to chess grandmasters, significantly expanding their understanding of the game."
Does this happen because we've stopped actively searching for better strategies? Or could it stem from our cognitive and biological limitations—or perhaps a prevailing belief that everything necessary to play at a grandmaster level has already been discovered?
This is another perfect example of how AI's non-human intelligence can surpass and complement human abilities. The fact that AI can generate strategies beyond human comprehension raises a key question: Do we stop innovating too soon? often settle for solutions that "work well enough." This happens for biological and cognitive limitations, the sunk-cost fallacy, or the belief that we've already discovered the best possible solution. Free from these constraints, AI can push past these limits and uncover new approaches. We've seen this in chess and fields like protein folding, where AI has solved problems humans have struggled with for decades.
That said, the challenge is to remain open to the fact that these AI-driven breakthroughs may force us to rethink our understanding of problems and even our role in solving them.
3. "The brain prepares the words we want to say before we actually say them, showing a forward-looking or predictive way of processing language. In contrast, when we listen, our brains make sense of the words slightly after we hear them, indicating a backward-looking way of understanding language. Our brains anticipate speech production but reactively process speech when listening."
Based on my limited knowledge of Biology, I think this idea applies to almost everything humans do. As you mentioned, the brain's predictive nature is at the core of how we interact with the world. Instead of passively reacting to external stimuli, the brain predicts what it will perceive based on past experiences and current context. It adjusts when there's a mismatch between its predictions and reality. This model applies to various aspects of our daily lives, including language, vision, and movement. This predictive model reduces the energy and time required for processing sensory information—critical for survival, as faster reactions increase the chances of avoiding danger or seizing opportunities.
However, AI doesn't need to mimic this survival and cognitive-limit-driven mechanism. Unlike humans, AI isn't constrained by evolutionary pressures like survival or reproduction and is not cognitively limited. It doesn't need to conserve energy or react quickly to threats. Instead, AI can focus on specific tasks, like solving problems or improving efficiency, without being limited by the biological constraints that shaped human cognition. While it's valuable to understand how human intelligence works, we shouldn't treat it as the only blueprint for solving problems.
This brings us to some of the more significant, more existential questions about AI that I've been thinking about—and I don't have definitive answers to them:
a) Can we fully understand how AI intelligence works with our cognitive capabilities?
Even today, many AI systems operate as "black boxes" where we struggle to interpret their internal processes. As AI advances toward AGI/ASI, this gap may only widen. While we can develop tools for explainability, there's no guarantee we'll fully comprehend the intelligence of a system vastly superior to our own.
b) Even if we understand AI intelligence, can we control it?
Understanding and control are two different things. Even if we grasp how an AGI or ASI operates, its capabilities might surpass our ability to impose meaningful control. This raises the importance of alignment research—ensuring that AI systems are aligned with human values and goals.
c) Will ASI show us how it thinks or works so we can control it?
If ASI is designed with transparency or interpretability in mind, it might be able to explain its reasoning and decision-making processes. However, once ASI surpasses human intelligence, it may not necessarily want to share how it works unless we ensure alignment with human goals and values from the outset. If ASI is aligned with human values, it might actively work to make its processes understandable to us. However, if alignment fails or the ASI develops its own goals, it might withhold this information or intentionally mislead us. There is more than 0% probability of the latter happening, and what we do to prepare for it?
d) Are we building our replacement?
This is the most profound and unsettling question. When AI reaches the ASI level, it may outcompete humans in every domain, potentially relegating us to a secondary role—or worse, treating us as irrelevant, much like humans have done to other species.
In my view, the ultimate goal of AI should not be to mimic human intelligence but to surpass human limitations and enhance our abilities. By prioritizing alignment, transparency, and collaboration, we can strive for a future where AI empowers humanity rather than poses a threat to it. However, this raises several critical questions:
a) Are we building AI that is truly aligned with human values? If so, are we ensuring that alignment serves the interests of all humanity rather than a select few?
b) Even if we believe that AI is human-aligned in the short run, will it be enough to prevent AI from eventually becoming our replacement?
c) What is our contingency plan if AI goes out of control, despite our best efforts to align it with human goals and ethics?
d) Who decides what "alignment with humanity" means?
e) How do we ensure accountability in AI systems?
Would an AI system understand: "With great power comes great responsibility." — Attributed to Voltaire
I will end again with B.F. Skinner quote:
"The real question is not whether machines think, but whether humans do."
Thank you MG, for another wonderful, detailed and thoughtful comment. Your points raise some of the most critical considerations surrounding AI development today.
Regarding your first point, I agree it's crucial to consider AI developing its own unique forms of intelligence. The AlphaZero study powerfully illustrates this, its success came not from mimicking humans, but from self-play leading to entirely non-human strategies. While understanding human cognition is valuable (as the Whisper/ECoG study attempts by mapping brain activity ), the goal shouldn't necessarily be replication but perhaps, as AlphaZero shows, augmentation or exploration of entirely different cognitive avenues unbound by our biological constraints. Your observation about predictive processing being potentially human-specific due to survival needs is pertinent here; AI might employ similar predictive mechanisms, as suggested by the Claude and Whisper studies, but for reasons tied to task optimization rather than evolutionary pressures, I think.
Your second point about AI overcoming human tendencies to stop innovating too soon is perfectly exemplified by the AlphaZero findings. The fact that novel, superior strategies existed in a game studied for centuries, and that grandmasters could learn and benefit from them, strongly suggests AI can push past human cognitive limits or accumulated dogma. It challenges us to remain open to fundamentally new approaches, as you say.
Finally, your reflections on the limits of our understanding, control, alignment, and the existential questions are incredibly valid. These are the deep challenges spurred by the very advancements these papers highlight. While studies like Anthropic's work on Claude's internals aim to increase transparency and understanding for safety and assessment, they represent early steps. Understanding the mechanism doesn't guarantee control, especially as complexity increases. The work described in the essay provides tools and insights that are necessary inputs for the vital discussions around alignment and accountability that you raise, but they don't resolve those questions themselves. The pursuit of interpretability, like that in the Claude paper, is partly motivated by the need to grapple with these very concerns about control and alignment.
Your concluding questions about defining alignment, ensuring accountability, and planning for contingencies are exactly where the conversation needs to head, informed by the kind of fundamental research explored in the essay. I am waiting for some reports from the recent UN meeting on AI control about alignment, because the labs cannot be relied upon - we MUST have government oversight and audit of these systems.
Thank you again for enriching the discussion with these critical points and the very fitting Skinner quote.
Can the government be relied on?
Before even having read the post, that graphic reminds of something I've been thinking about for a very long time now. It's been my casual observation that thought is an abstract - and rather amorphous - thing.
We have a thought -> convert it into language/words -> speak those words -> listener hears the words -> converts the words back into their own abstract thought.
//
Thus, there seem to be two points of potential misinterpretation. The first one when we try to formulate the abstract thought into words, with varying degrees of imperfection. The second when the listener converts the heard words back into their own abstract thought, also with varying degrees of imperfection.
That is incredibly astute - your observation perfectly frames one of the core challenges these studies grapple with: how abstract representations or intentions (whether in human brains or AI models) get translated into and out of the concrete form of language, and how we might develop tools to better understand and bridge those potentially 'lossy' translation steps.
Your point about the "varying degrees of imperfection" in formulating abstract thought into words is exactly what makes studying AI internals so difficult, yet fascinating. The Anthropic paper tries to map Claude's internal, abstract "features" and their interactions (its "thought processes," in a sense) onto its specific linguistic output. The methods like "circuit tracing" are attempts to make that first conversion step, from the internal state to the output, less opaque, though as I show in my essay, it's incredibly complex and the understanding is still imperfect.
Mapping Thought-to-Speech and Speech-to-Thought (Whisper/ECoG paper): The Nature Human Behaviour study directly investigates the neural basis of the two conversion points you mentioned.
It maps brain activity during the "convert thought into language/words -> speak" process (production) and the "listener hears words -> converts words back into thought" process (comprehension).
Interestingly, it finds that continuous, high-dimensional AI embeddings (representing acoustic, speech, and language information) map remarkably well onto brain activity, perhaps better than traditional, discrete linguistic units like phonemes or parts-of-speech. This could suggest that the brain's internal "abstract thought" related to language might also be better represented as a complex, high-dimensional pattern rather than easily captured by discrete words, aligning with your idea of thought being "abstract - and rather amorphous." The study literally traces the flow from internal language representations to speech output before word onset, and from auditory input to language representations after word onset.
Translating Abstract Concepts (AlphaZero paper): The AlphaZero research tackles a related problem: translating the AI's abstract, internally derived strategic concepts into something humans can understand and learn from. Using chess puzzles as specific examples ("prototypes") is their way of converting AlphaZero's abstract vector representations into a concrete form for human interpretation, which again speaks to the challenge of that first conversion step you identified.
You are onto something!
I tinkered with the Markov model when I took a Coursera course in Java. I wasn't really interested in pursuing AI, I just wanted to get the Java badge, but it was an interesting peek at the foundation of LLM's. Of course, modern day LLM's are a helluva lot more sophisticated than the simple two dimensional array that I was using to track words and their most likely follow ups. Nothing like ChatGPT or any of these other modern LLM's.
I can only assume they use some kind of machine learning/neural network, which, yeah, will complicate tracing the process. I can only imagine the size of the log if every node records it's current state, how the process arrived there, where it will send the process next, etc., and then trying to read that log and make sense of it, but I can't imagine any other way to pull it off.
Part of why I have been writing about aphantasia is not just because I have it but also because AI has it, in its way. Its unconscious (such as it is) is structured in a language, like a language, with language. It understands the future not by visualizing but by what language says about the past... https://hollisrobbinsanecdotal.substack.com/p/aphantasia-and-mental-modeling
That is a great connection that you have uncovered Hollis. It should be shared with researchers at Anthropic and Google DeepMind - maybe Anthropic connected with your piece via Tyler, I will see if I can get their thoughts, I can also share with Shane Legg at DeepMind to see what they have on aphantasia? Shall I do that?
Yes! I would love to see some real focus on on this…. Thank you!
Great, I will see what I can do, will let you know