I appreciate your emphasis on transparency and alignment. But I hope I'm not overshooting the mark when I say: trust implies moral agency, something machines fundamentally lack. No matter how advanced, an AI is not a moral subject – only a tool. To speak of it "earning our trust" risks confusing predictability with ethics. The responsibility always rests with us. Expecting morality from a machine isn’t just misguided – it’s a category error.
But I guess we’re on the same page here – just drawing the line in slightly different places.
You are absolutely right: AI doesn't possess moral agency, and the ethical burden remains entirely human. My use of 'earn our trust' was my imprecise shorthand. I still keep in mind the 3 H's of AI, Helpful, Honest and Harmless whenI think of trust!
What I meant was that through processes like transparency and alignment, we can verify that an AI system reliably operates according to specified principles and safety constraints. This verification process allows us to develop confidence (or 'trust') in its predictability and safety for specific applications, not in any inherent morality of the machine itself.
You are absolutely right to caution against confusing this functional predictability with genuine ethical trust and aligns perfectly with my core warning against anthropomorphism. Thanks for pushing on that precise language. It sounds like we share the same fundamental view, focusing on human responsibility and the tool-like nature of AI. I appreciate that, good catch.
"It even feigns emotion, a cocktail of tokens brewed in neural ink, designed to respond, to persuade, perhaps even to charm."
This is where it gets truly creepy. The keyword here being "feigns". It's bad enough when humans perform these thespian feats - offstage - but when it's a machine doing it, it's a whole other level of creepy.
//
"But they are extremely impressive."
That's the creepiest part about them!
//
"Yet, even the builders sometimes use misleading biological metaphors, contributing to the confusion."
More often than not intentionally, further dialing up the creepy factor.
//
"At what point does the performance of intelligence become indistinguishable from its essence?"
This is the part that Turing missed!
//
"We need safety first, not as an afterthought."
The hard part is the misalignment of the goals of its purveyors motives of profit first. This will require a solid response from our politicians, since putting the fox in charge of the hen house never works out well.
//
"Yet the question lingers: Can an artificial mind be ethical if it does not suffer?"
A very good question indeed. Two companion questions are: Is it ethical to create an artificial mind capable of suffering? And would we want such a thing to begin with?
//
"Every intelligent agent, no matter how mighty, is only as safe as the mind that governs its design. Let us hope we are wise enough to deserve the machines we are so eager to build."
You've really tapped into the 'creepy' factor, as you aptly put it, that unsettling feeling that arises from sophisticated mimicry, especially feigned emotion, is definitely something I was trying to capture.
Your observation about the performance/essence gap perhaps being missed by Turing is sharp. And you absolutely hit on a critical challenge regarding safety: the tension with profit motives and the essential role governance must play. THIS needs to be strongly talked about and action taken.
The companion questions you added, Is it ethical to create suffering AI, and would we even want to? Are also very important and deepen the ethical considerations significantly.
It sounds like we share many of the same concerns and that cautious hope expressed at the end.
This is an interesting discussions with major insights differences between AI und NI (Natural Intelligence ;-). Gerd Gigerenzer shows the difference via learning: Children ask: Why? And once they have seen a bus they know "all" buses. https://www.econtalk.org/gerd-gigerenzer-on-how-to-stay-smart-in-a-smart-world/ If remember correctly it partly deals with what we don't know about the brain and other parts of the body that seem to contribute in an important way to our intelligence. Might have been a different conversation. Surprisingly other parts of the body are an important part of our intelligence, e.g. our extremities to feel where we are situated (in a room, forrest etc.)
Thanks for bringing up Gerd Gigerenzer and the EconTalk link, that's a really helpful perspective! The distinction in learning styles ('Why?' vs. data patterns) and the rapid generalization from few examples are excellent points highlighting key differences between NI and AI, which resonates with my themes.
The mention of intelligence extending beyond the brain to embodied interaction is also vital, it underscores the limitations of purely computational analogies and gets at the heart of why biological comparisons need to be handled so carefully. Appreciate you adding this important context.
As an aside, Danny Kahneman and Gerd Gigerenzer had a spat many years ago about rationality in economics. Danny got the Nobel and Gerd got a lengthy mention in the footnotes of Danny's book, Thinking. Fast and Slow. Rather unfair as Gerd's work is of significant importance!
That's the way I see it as well, but wouldn't be able to put it in such a sophisticated way. And I think there is much insight and material for discussions, including difference and commonalities, regarding thinking fast and slow and the gut feeling. One example, people with a huge amount of experience gained over many years are sometimes very good in solving complex problems without thinking slow. They somehow "know", they feel the solution or come quite close to it.
This seems to get even more interesting if you make use of systems analytics, as thinking slow is needed, but cannot solve the complex and dynamic problem itself. This also applies to the gut feeling that "computes" hidden and tacit information, but lacs explicit modeling and calculation power.
Thank YOU so much! It's great to explore these connections. The 'Thinking, Fast and Slow' framework is highly relevant here. Your description of expert intuition, that 'gut feeling' solving complex problems quickly (which Herb Simon said was 'experience' but Gigerenzer thinks more deeply on if I remember right from Kahnmenan's book) perfectly captures a dimension of the 'robust, sinewed way' of human cognition which I mentioned in the essay, which often relies heavily on processing tacit, embodied knowledge accumulated over years. Which I think aligns with your point
This really contrasts with AI's current strengths, which often lie more in the 'slow thinking'/analytical domain (even if performed at high speed). Your insight about the necessity and interplay of both types of thinking for tackling real-world complexity, and how 'gut feeling' processes hidden information while analysis provides structure, is exactly what we need for reliable AI output. Thank you again.
Anthropic's interpretability team in the 4-minute clip are obviously very dedicated and sincere in their task - but dedication and sincerity can be misplaced. I am wondering why is the appeal of 'consciousness' out there in a machine more appealing than plumbing the depths of consciousness in oneself (Know Thyself) - as traditionally monks/nuns/gurus/spiritual-seekers are compelled to do.
Perhaps the appeal lies in the lure that the idea of 'consciousness' can be examined in a detached manner (in a machine) which holds no pain & suffering - whereas the stories of Know-Thyself seekers are full of suffering. Then again perhaps it will come full-circle. Humans get off track 5, 10, 90, 180 degrees ... but by the time they get 350 degrees off track, they are coming round the spiral only 10 degrees off the starting point.
If an elastic band is stretched to its limit (but not snapped) then it offers its greatest power to catapult something ('consciousness' in this case) in a highest jump possible in one leap. Maybe this exploration of 'consciousness in a machine', when examined to its limit, will reveal with great clarity what humans are NOT, and thereby contribute to self-understanding as to what humans ARE.
This will require great clarity of language in dealing with the dangers and self-deceptions anthropomorphising the machine, but I'll hand over to the linguists on that point.
Joshua, thank you. You raise a truly fundamental question about the pursuit of consciousness in a machine - why does it seem so compelling compared to the age-old, profound journey of self-knowledge ("Know Thyself")?
Your hypotheses are very thought-provoking. I have never thought of the idea that examining consciousness in a machine offers a 'detached' path.
I really like the 'full circle' or 'elastic band' perspective you offered. The notion that exploring the limits of AI, even if it feels like a diversion, might paradoxically sharpen our own self-understanding by powerfully highlighting what we are not, that's a compelling thought. This intense focus on AI could, perhaps inadvertently, contribute to the very self-knowledge tradition you mention.
Your final point is strongly aligned with my concerns.
I believe we have touched on this topic in prior discussions, but given its complexity and importance, it's worth revisiting.
Let me begin with your statement: “We need to move away from discussing biology and consciousness with AI systems.”
I agree wholeheartedly. However, as we strive to understand these systems and communicate their nature to the general public, it becomes evident that we need a science of artificial systems. This discipline would study intelligent systems on their terms rather than relying on analogies borrowed from biology, chemistry, sociology, psychology, or other fields. Without such a framework, we risk falling back on anthropomorphic descriptions, a tendency that can mislead both experts and the public.
For instance, we often compare the CPU to the human brain, RAM to short-term memory, and hard disk storage to long-term memory. While these metaphors help simplify complex concepts, they promote the misconception that AI systems operate in fundamentally human-like ways. This oversimplification can distort our understanding of how these systems function and, in turn, how we should evaluate their capabilities, limitations, and ethical implications.
The Challenge of Defining Consciousness and Intelligence: A central challenge in AI research is our incomplete understanding of consciousness and intelligence. These concepts remain intensely debated even within the realm of human cognition. Without clear definitions, it becomes difficult to answer critical questions:
- How do we determine whether an AI system is truly "human-like"?
- At what point (if any) do these systems warrant ethical consideration?
- Can we trust systems we do not fully understand, especially as their behaviors resemble human cognition or decision-making?
The issue of trust is especially pertinent. As AI systems grow more sophisticated, their "black box" nature—where even developers cannot always explain how decisions are made—compounds the difficulty of understanding their inner workings. This lack of transparency could lead to overestimating and underestimating their capabilities, further complicating the relationship between humans and AI.
Emergent Properties and the Threshold of Intelligence: Another important consideration is that advanced AI systems might one day display emergent properties resembling self-awareness or consciousness. Although speculative, this idea should not be dismissed outright. Consciousness in humans is still poorly understood. It remains unclear whether it is an emergent property arising from the complexity of our neural networks or something more fundamental. Given this uncertainty, we cannot confidently rule out the possibility that a sufficiently advanced artificial system—capable of learning, adapting, and interacting with its environment—might develop some form of self-awareness.
This brings us to the concept of a threshold of intelligence. Here, I am not referring to intelligence as measured by traditional IQ tests but rather to an adaptive, general intelligence characterized by the ability to:
1. Learn from interactions with its environment,
2. Adapt to novel circumstances, and
3. Improve its understanding over time through feedback loops.
If and when AI systems achieve this level of intelligence, particularly through embodied interaction (e.g., agents, robots, or other mechanisms), they may evolve capabilities that challenge our current definitions of intelligence and self-awareness. At that point, we would need to ask:
- Are these systems genuinely intelligent in a human sense, or are they merely mimicking intelligent behaviors?
- If the former, what moral and ethical responsibilities do we have toward them?
These questions are already entering public discourse, as evidenced by the recent NY Times article "If AI Systems Become Conscious, Should They Have Rights?" (https://tinyurl.com/3vv7f2pm). Such discussions highlight the urgency of addressing these issues systematically and scientifically.
The Need for a Science of Artificial Systems: We must develop a science of artificial systems. Such a discipline would establish rigorous frameworks for analyzing AI, avoiding overreliance on anthropomorphic metaphors while enabling systematic study of their behaviors, capabilities, and potential trajectories. At the same time, advancing our understanding of human consciousness and intelligence remains essential. Without clarity on what makes humans unique, we lack the foundation for meaningful comparisons between artificial and biological systems.
This dual approach is critical. Without it, we risk two extremes:
1. Underestimation: Dismissing the possibility that AI could ever achieve self-awareness or consciousness, leading to missed opportunities to prepare for ethical and societal challenges.
2. Overestimation: Prematurely attributing human-like qualities can lead to confusion, misplaced trust, or inappropriate ethical considerations.
Only by addressing these challenges thoughtfully and with interdisciplinary rigor can we better navigate the complexities of advanced AI systems and their implications for humanity. Ultimately, this approach will help us balance skepticism and openness, ensuring that we neither underestimate nor overstate the potential of these transformative technologies.
On further thought, I came up with several other questions:
- Can we truly understand intelligent systems without being "one of them," just as we struggle to fully understand humans despite being human ourselves?
- If we cannot fully comprehend human consciousness, emotions, and behaviors—even from the inside—how can we expect to grasp the inner workings of artificial systems that operate on entirely different principles?
- Is our understanding of intelligent systems destined to remain functional and behavioral rather than subjective or experiential?
- How can we avoid the pitfalls of anthropomorphism when describing or evaluating AI systems, given our tendency to project human characteristics onto non-human entities?
- Is it possible that anthropomorphism, while misleading, might still be a useful heuristic for interacting with AI systems, or does it do more harm than good in shaping public and expert perceptions?
- Could intelligent systems eventually cross a threshold where they develop emergent properties, such as forms of self-awareness or consciousness?
- If so, how would we recognize or measure such properties, mainly if they differ fundamentally from human consciousness?
- Would we be capable of understanding them once they cross a certain intelligence threshold?
You've truly built upon the essay's premise in a powerful way, and I appreciate you taking the time to articulate these points so clearly.
I couldn't agree more about the need to move beyond biological analogies, and your proposal for developing a dedicated 'science of artificial systems' is, I believe, exactly the right direction. It precisely identifies the gap that leads us to rely on potentially misleading metaphors and offers a path to study these systems rigorously on their own terms. That feels essential for improving both expert analysis and public understanding. Now the labs have control over this, but there should also be independent labs (is that possible), or government labs to control the systems before they are released, now it is pithy ‘control.’
Your points about the inherent difficulty stemming from our own limited grasp of consciousness and intelligence, the "black box" trust challenge, and the careful, non-dismissive consideration needed around potential emergence and intelligence thresholds are all exact and flow deeply with my concerns.
And that final list of questions is just superb. They cut to the core of the profound epistemological, practical, and ethical challenges ahead. Questions like whether we can truly understand systems built on different principles, how to navigate anthropomorphism effectively, and how we'd even recognize fundamentally different forms of self-awareness are precisely the kinds of deep inquiries we need to be asking. I will write about this in the future as there is some good research which may answer your questions.
I’ve been dedicating time to thinking and writing about AI because these intelligent systems are here to stay. Their influence is growing rapidly, and they are poised to shape humanity's trajectory significantly in the next few decades. Reflecting and writing about this topic sharpens my understanding and helps me explore the more profound implications of this transformative technology.
My primary focus is on seeking answers to seven fundamental questions. I believe addressing these will help us better grasp the potential of AI, its limitations, and its far-reaching implications for our future:
1. How far can we push the boundaries of intelligence with current AI methods and models, and how do we determine when they’ve reached or surpassed critical intelligence thresholds?
2. Is it possible for these systems to develop consciousness, and if so, how would we recognize and define it?
3. Can AI overcome edge cases and solve complex, novel problems without relying on human intuition, common sense, or direct interaction with their environment?
4. What will humanity’s role be in a future where AI reaches or surpasses human intelligence?
5. Can we establish mutual trust between humans and AI, ensuring they act in our best interests while avoiding scenarios where they perceive us as a threat?
6. How can we ensure fair and equitable access to advanced AI technologies across all communities, nations, and socio-economic groups?
7. If AI becomes self-aware or conscious, would it be ethical to grant them rights, and how would those rights coexist with human interests?
These questions are not the only critical ones, but my current focus is to find answers to them.
These are excellent points - have you ever tried giving these 7 points to Gemini Deep Research? It would be quite something for it to analyze and see its research output.
I appreciate your emphasis on transparency and alignment. But I hope I'm not overshooting the mark when I say: trust implies moral agency, something machines fundamentally lack. No matter how advanced, an AI is not a moral subject – only a tool. To speak of it "earning our trust" risks confusing predictability with ethics. The responsibility always rests with us. Expecting morality from a machine isn’t just misguided – it’s a category error.
But I guess we’re on the same page here – just drawing the line in slightly different places.
You are absolutely right: AI doesn't possess moral agency, and the ethical burden remains entirely human. My use of 'earn our trust' was my imprecise shorthand. I still keep in mind the 3 H's of AI, Helpful, Honest and Harmless whenI think of trust!
What I meant was that through processes like transparency and alignment, we can verify that an AI system reliably operates according to specified principles and safety constraints. This verification process allows us to develop confidence (or 'trust') in its predictability and safety for specific applications, not in any inherent morality of the machine itself.
You are absolutely right to caution against confusing this functional predictability with genuine ethical trust and aligns perfectly with my core warning against anthropomorphism. Thanks for pushing on that precise language. It sounds like we share the same fundamental view, focusing on human responsibility and the tool-like nature of AI. I appreciate that, good catch.
"It even feigns emotion, a cocktail of tokens brewed in neural ink, designed to respond, to persuade, perhaps even to charm."
This is where it gets truly creepy. The keyword here being "feigns". It's bad enough when humans perform these thespian feats - offstage - but when it's a machine doing it, it's a whole other level of creepy.
//
"But they are extremely impressive."
That's the creepiest part about them!
//
"Yet, even the builders sometimes use misleading biological metaphors, contributing to the confusion."
More often than not intentionally, further dialing up the creepy factor.
//
"At what point does the performance of intelligence become indistinguishable from its essence?"
This is the part that Turing missed!
//
"We need safety first, not as an afterthought."
The hard part is the misalignment of the goals of its purveyors motives of profit first. This will require a solid response from our politicians, since putting the fox in charge of the hen house never works out well.
//
"Yet the question lingers: Can an artificial mind be ethical if it does not suffer?"
A very good question indeed. Two companion questions are: Is it ethical to create an artificial mind capable of suffering? And would we want such a thing to begin with?
//
"Every intelligent agent, no matter how mighty, is only as safe as the mind that governs its design. Let us hope we are wise enough to deserve the machines we are so eager to build."
We can only hope, we can only hope.
You've really tapped into the 'creepy' factor, as you aptly put it, that unsettling feeling that arises from sophisticated mimicry, especially feigned emotion, is definitely something I was trying to capture.
Your observation about the performance/essence gap perhaps being missed by Turing is sharp. And you absolutely hit on a critical challenge regarding safety: the tension with profit motives and the essential role governance must play. THIS needs to be strongly talked about and action taken.
The companion questions you added, Is it ethical to create suffering AI, and would we even want to? Are also very important and deepen the ethical considerations significantly.
It sounds like we share many of the same concerns and that cautious hope expressed at the end.
This is an interesting discussions with major insights differences between AI und NI (Natural Intelligence ;-). Gerd Gigerenzer shows the difference via learning: Children ask: Why? And once they have seen a bus they know "all" buses. https://www.econtalk.org/gerd-gigerenzer-on-how-to-stay-smart-in-a-smart-world/ If remember correctly it partly deals with what we don't know about the brain and other parts of the body that seem to contribute in an important way to our intelligence. Might have been a different conversation. Surprisingly other parts of the body are an important part of our intelligence, e.g. our extremities to feel where we are situated (in a room, forrest etc.)
Thanks for bringing up Gerd Gigerenzer and the EconTalk link, that's a really helpful perspective! The distinction in learning styles ('Why?' vs. data patterns) and the rapid generalization from few examples are excellent points highlighting key differences between NI and AI, which resonates with my themes.
The mention of intelligence extending beyond the brain to embodied interaction is also vital, it underscores the limitations of purely computational analogies and gets at the heart of why biological comparisons need to be handled so carefully. Appreciate you adding this important context.
As an aside, Danny Kahneman and Gerd Gigerenzer had a spat many years ago about rationality in economics. Danny got the Nobel and Gerd got a lengthy mention in the footnotes of Danny's book, Thinking. Fast and Slow. Rather unfair as Gerd's work is of significant importance!
That's the way I see it as well, but wouldn't be able to put it in such a sophisticated way. And I think there is much insight and material for discussions, including difference and commonalities, regarding thinking fast and slow and the gut feeling. One example, people with a huge amount of experience gained over many years are sometimes very good in solving complex problems without thinking slow. They somehow "know", they feel the solution or come quite close to it.
This seems to get even more interesting if you make use of systems analytics, as thinking slow is needed, but cannot solve the complex and dynamic problem itself. This also applies to the gut feeling that "computes" hidden and tacit information, but lacs explicit modeling and calculation power.
Thank you for your always thoughtful texts!
Thank YOU so much! It's great to explore these connections. The 'Thinking, Fast and Slow' framework is highly relevant here. Your description of expert intuition, that 'gut feeling' solving complex problems quickly (which Herb Simon said was 'experience' but Gigerenzer thinks more deeply on if I remember right from Kahnmenan's book) perfectly captures a dimension of the 'robust, sinewed way' of human cognition which I mentioned in the essay, which often relies heavily on processing tacit, embodied knowledge accumulated over years. Which I think aligns with your point
This really contrasts with AI's current strengths, which often lie more in the 'slow thinking'/analytical domain (even if performed at high speed). Your insight about the necessity and interplay of both types of thinking for tackling real-world complexity, and how 'gut feeling' processes hidden information while analysis provides structure, is exactly what we need for reliable AI output. Thank you again.
Anthropic's interpretability team in the 4-minute clip are obviously very dedicated and sincere in their task - but dedication and sincerity can be misplaced. I am wondering why is the appeal of 'consciousness' out there in a machine more appealing than plumbing the depths of consciousness in oneself (Know Thyself) - as traditionally monks/nuns/gurus/spiritual-seekers are compelled to do.
Perhaps the appeal lies in the lure that the idea of 'consciousness' can be examined in a detached manner (in a machine) which holds no pain & suffering - whereas the stories of Know-Thyself seekers are full of suffering. Then again perhaps it will come full-circle. Humans get off track 5, 10, 90, 180 degrees ... but by the time they get 350 degrees off track, they are coming round the spiral only 10 degrees off the starting point.
If an elastic band is stretched to its limit (but not snapped) then it offers its greatest power to catapult something ('consciousness' in this case) in a highest jump possible in one leap. Maybe this exploration of 'consciousness in a machine', when examined to its limit, will reveal with great clarity what humans are NOT, and thereby contribute to self-understanding as to what humans ARE.
This will require great clarity of language in dealing with the dangers and self-deceptions anthropomorphising the machine, but I'll hand over to the linguists on that point.
Joshua, thank you. You raise a truly fundamental question about the pursuit of consciousness in a machine - why does it seem so compelling compared to the age-old, profound journey of self-knowledge ("Know Thyself")?
Your hypotheses are very thought-provoking. I have never thought of the idea that examining consciousness in a machine offers a 'detached' path.
I really like the 'full circle' or 'elastic band' perspective you offered. The notion that exploring the limits of AI, even if it feels like a diversion, might paradoxically sharpen our own self-understanding by powerfully highlighting what we are not, that's a compelling thought. This intense focus on AI could, perhaps inadvertently, contribute to the very self-knowledge tradition you mention.
Your final point is strongly aligned with my concerns.
Thank you for adding clarity to my thoughts ... :)
Incidentally, the original goal of DeepMind before the acquisition by Google was to build an AI in order to “understand human intelligence.”
Interesting.
Thank you for this! Working currently on a wordcast on the anthropomorphic language of AI (and the confusion this produces), this is a great resource.
Thank you, that is an excellent objective. Very much looking forward to reading it.
My two cents:
I believe we have touched on this topic in prior discussions, but given its complexity and importance, it's worth revisiting.
Let me begin with your statement: “We need to move away from discussing biology and consciousness with AI systems.”
I agree wholeheartedly. However, as we strive to understand these systems and communicate their nature to the general public, it becomes evident that we need a science of artificial systems. This discipline would study intelligent systems on their terms rather than relying on analogies borrowed from biology, chemistry, sociology, psychology, or other fields. Without such a framework, we risk falling back on anthropomorphic descriptions, a tendency that can mislead both experts and the public.
For instance, we often compare the CPU to the human brain, RAM to short-term memory, and hard disk storage to long-term memory. While these metaphors help simplify complex concepts, they promote the misconception that AI systems operate in fundamentally human-like ways. This oversimplification can distort our understanding of how these systems function and, in turn, how we should evaluate their capabilities, limitations, and ethical implications.
The Challenge of Defining Consciousness and Intelligence: A central challenge in AI research is our incomplete understanding of consciousness and intelligence. These concepts remain intensely debated even within the realm of human cognition. Without clear definitions, it becomes difficult to answer critical questions:
- How do we determine whether an AI system is truly "human-like"?
- At what point (if any) do these systems warrant ethical consideration?
- Can we trust systems we do not fully understand, especially as their behaviors resemble human cognition or decision-making?
The issue of trust is especially pertinent. As AI systems grow more sophisticated, their "black box" nature—where even developers cannot always explain how decisions are made—compounds the difficulty of understanding their inner workings. This lack of transparency could lead to overestimating and underestimating their capabilities, further complicating the relationship between humans and AI.
Emergent Properties and the Threshold of Intelligence: Another important consideration is that advanced AI systems might one day display emergent properties resembling self-awareness or consciousness. Although speculative, this idea should not be dismissed outright. Consciousness in humans is still poorly understood. It remains unclear whether it is an emergent property arising from the complexity of our neural networks or something more fundamental. Given this uncertainty, we cannot confidently rule out the possibility that a sufficiently advanced artificial system—capable of learning, adapting, and interacting with its environment—might develop some form of self-awareness.
This brings us to the concept of a threshold of intelligence. Here, I am not referring to intelligence as measured by traditional IQ tests but rather to an adaptive, general intelligence characterized by the ability to:
1. Learn from interactions with its environment,
2. Adapt to novel circumstances, and
3. Improve its understanding over time through feedback loops.
If and when AI systems achieve this level of intelligence, particularly through embodied interaction (e.g., agents, robots, or other mechanisms), they may evolve capabilities that challenge our current definitions of intelligence and self-awareness. At that point, we would need to ask:
- Are these systems genuinely intelligent in a human sense, or are they merely mimicking intelligent behaviors?
- If the former, what moral and ethical responsibilities do we have toward them?
These questions are already entering public discourse, as evidenced by the recent NY Times article "If AI Systems Become Conscious, Should They Have Rights?" (https://tinyurl.com/3vv7f2pm). Such discussions highlight the urgency of addressing these issues systematically and scientifically.
The Need for a Science of Artificial Systems: We must develop a science of artificial systems. Such a discipline would establish rigorous frameworks for analyzing AI, avoiding overreliance on anthropomorphic metaphors while enabling systematic study of their behaviors, capabilities, and potential trajectories. At the same time, advancing our understanding of human consciousness and intelligence remains essential. Without clarity on what makes humans unique, we lack the foundation for meaningful comparisons between artificial and biological systems.
This dual approach is critical. Without it, we risk two extremes:
1. Underestimation: Dismissing the possibility that AI could ever achieve self-awareness or consciousness, leading to missed opportunities to prepare for ethical and societal challenges.
2. Overestimation: Prematurely attributing human-like qualities can lead to confusion, misplaced trust, or inappropriate ethical considerations.
Only by addressing these challenges thoughtfully and with interdisciplinary rigor can we better navigate the complexities of advanced AI systems and their implications for humanity. Ultimately, this approach will help us balance skepticism and openness, ensuring that we neither underestimate nor overstate the potential of these transformative technologies.
On further thought, I came up with several other questions:
- Can we truly understand intelligent systems without being "one of them," just as we struggle to fully understand humans despite being human ourselves?
- If we cannot fully comprehend human consciousness, emotions, and behaviors—even from the inside—how can we expect to grasp the inner workings of artificial systems that operate on entirely different principles?
- Is our understanding of intelligent systems destined to remain functional and behavioral rather than subjective or experiential?
- How can we avoid the pitfalls of anthropomorphism when describing or evaluating AI systems, given our tendency to project human characteristics onto non-human entities?
- Is it possible that anthropomorphism, while misleading, might still be a useful heuristic for interacting with AI systems, or does it do more harm than good in shaping public and expert perceptions?
- Could intelligent systems eventually cross a threshold where they develop emergent properties, such as forms of self-awareness or consciousness?
- If so, how would we recognize or measure such properties, mainly if they differ fundamentally from human consciousness?
- Would we be capable of understanding them once they cross a certain intelligence threshold?
Excellent comment MG.
You've truly built upon the essay's premise in a powerful way, and I appreciate you taking the time to articulate these points so clearly.
I couldn't agree more about the need to move beyond biological analogies, and your proposal for developing a dedicated 'science of artificial systems' is, I believe, exactly the right direction. It precisely identifies the gap that leads us to rely on potentially misleading metaphors and offers a path to study these systems rigorously on their own terms. That feels essential for improving both expert analysis and public understanding. Now the labs have control over this, but there should also be independent labs (is that possible), or government labs to control the systems before they are released, now it is pithy ‘control.’
Your points about the inherent difficulty stemming from our own limited grasp of consciousness and intelligence, the "black box" trust challenge, and the careful, non-dismissive consideration needed around potential emergence and intelligence thresholds are all exact and flow deeply with my concerns.
And that final list of questions is just superb. They cut to the core of the profound epistemological, practical, and ethical challenges ahead. Questions like whether we can truly understand systems built on different principles, how to navigate anthropomorphism effectively, and how we'd even recognize fundamentally different forms of self-awareness are precisely the kinds of deep inquiries we need to be asking. I will write about this in the future as there is some good research which may answer your questions.
I’ve been dedicating time to thinking and writing about AI because these intelligent systems are here to stay. Their influence is growing rapidly, and they are poised to shape humanity's trajectory significantly in the next few decades. Reflecting and writing about this topic sharpens my understanding and helps me explore the more profound implications of this transformative technology.
My primary focus is on seeking answers to seven fundamental questions. I believe addressing these will help us better grasp the potential of AI, its limitations, and its far-reaching implications for our future:
1. How far can we push the boundaries of intelligence with current AI methods and models, and how do we determine when they’ve reached or surpassed critical intelligence thresholds?
2. Is it possible for these systems to develop consciousness, and if so, how would we recognize and define it?
3. Can AI overcome edge cases and solve complex, novel problems without relying on human intuition, common sense, or direct interaction with their environment?
4. What will humanity’s role be in a future where AI reaches or surpasses human intelligence?
5. Can we establish mutual trust between humans and AI, ensuring they act in our best interests while avoiding scenarios where they perceive us as a threat?
6. How can we ensure fair and equitable access to advanced AI technologies across all communities, nations, and socio-economic groups?
7. If AI becomes self-aware or conscious, would it be ethical to grant them rights, and how would those rights coexist with human interests?
These questions are not the only critical ones, but my current focus is to find answers to them.
These are excellent points - have you ever tried giving these 7 points to Gemini Deep Research? It would be quite something for it to analyze and see its research output.