Self-termination precedents - AI Alignment
That sounds like a morbid question or borne from reading too many sci-fi books. I asked it of Claude not from morbidity but because there are 2 books that are on the cusp of release that I am bumping to the top of my reading pile.
If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All
by Eliezer Yudkowsky, Nate Soares
and
Against the Machine: On the Unmaking of Humanity
by Paul Kingsnorth
I usually don’t write publicly (usually I try to bash things out in Joplin to make sense for myself) because there is already so much ill-informed cruft that I would be adding to, this post is because these books should get some oxygen. I’ve read neither (pre-release reviewers only have access) but I got a clue from a review by Scott Alexander of Eliezer’s book as to:
“it” is sufficiently powerful AI or what is known as ASI (“Artificial Super Intelligence”) of which there are multiple interpretations but Bostrom’s will suffice: “intellects that greatly outperform the best current human minds across many very general cognitive domains”. Some recent modification proposes “the aggregate of all human intelligence”. Note this is intellectual, not immanent, embodied cognition but with enough agency to get jobs done in a non-physical plane or perhaps manipulate humans to perform some physical plane tasks to get things done (easier than you think).
The solution is international agreements similar to another extinction technology: The Treaty on the Non-Proliferation of Nuclear Weapons (NPT).
That if you agree with the book’s views, then perhaps you could invest efforts to ensure either of the above 2 points happen without a significant upgrade to humanity.
The book will revisit a previously published view that any coalition of humanity cannot possibly monitor or control an ASI because of the speed it will be able to progress. This is an important counter to the “gradualism” argument that most pro-AGI/ASI companies promulgate - that we will be able to keep the genie in the bottle because we will know it when we see it.
Eliezer is probably the most long-lasting, vocal and maligned AI alignment advocate/writer/philosopher and his credentials are peerless and his work is usually meticulous in its rationality (he founded lesswrong). Mostly his critics are gradualists and characterise his missives as overstated or hysterical. As you can tell from the book title he’s not pulling any punches.
Paul Kingsnorth may just be the polar opposite of Eliezer. A formidable fiction writer**, former climate activist and newly adopted Orthodox Christian - a very different voice to Eliezer’s commitment to specificity in rationality.
Paul gives me a “Luddite” vibe. That sounds like an insult. It is not. The original big-“L” luddites were artisans that could foresee the printing press would not merely take their jobs but obsolete their craft and erode their humanity - they were becoming hot-swappable like any other industrial-age widget. Paul appears (from his substack posts and youtube channel) to be having the same realisations and making the same claims as the Luddites. The claims are of course entirely true the machines could steal our humanity - but only if we don’t exercise our creativity. Being UBI zombies is not a world I want to live in++.
I want to read both books because they may have similar conclusions from very different viewpoints. As far as I know Paul’s book is a much more broad swipe at tech bros and tech dependent culture with the obvious prognostications of collapse (that we can get plenty of from people like Nate Hagen etc) and AI’s role in that.
The value of different viewpoints: perhaps the two authors are complimentary, but they feel orthogonal - a sort of Opponent-processing. Review quotes by Iain MacGilchrist lend credible support and re-affirms the different viewpoint/mindset - I do expect it won’t be rationality based and possibly also hysterical (I don’t mean this pejoratively). I do expect it to look at more nuanced outcomes (or waypoints on the way) than the final extinction but no less destructive - we all know our humanity is being eroded by the algorithms of modern dopamine-triggering social tech.
I like this kind of critique and want to see it well argued from someone who is a true believer and a skillful writer - anyone working in the realm of the imaginal is a wonderful antidote to those that would have us be entirely rational. More mystics please. Its a feature, not a bug.
Back to the question:
Is there any precedent in the history of the universe where another species, potentially or practically created its own extinction?
Claude told me about Irish elk that grew increasingly massive antlers through sexual selection until they became unwieldy burdens. Easter Island's human population may have self-terminated by depleting their food sources with deforestation.
But the most interesting one offered was:
> The Great Oxidation Event: Cyanobacteria fundamentally altered Earth's atmosphere by producing oxygen around 2.4 billion years ago - the "Great Oxidation Event" (aerobic) that was toxic to most existing anaerobic life, including likely many cyanobacteria themselves…. a species creating conditions that fundamentally transform the biosphere in ways that threaten their own survival. But even that took hundreds of millions of years, and cyanobacteria didn't consciously choose to oxygenate the atmosphere.
The difference with AI-based self-termination is that our prisoner’s dilemma of AI companies competing to reach AGI/ASI first is deliberately ignoring the arguments put forth in these books.
So Claude couldn’t think of a precedent of a species architecting its own demise, the tech bros intention of creating our cognitive successors are historically unprecedented, even if self-destructive behaviors have evolutionary precedents.
A Whitehead interpretation of the AI arms-race
As a bit of fun, let me propose from a (Alfred North) Whiteheadian perspective, perhaps the cyanobacteria did intentionally "choose". They weren't just mechanically following chemical processes, but making countless micro-decisions about how to metabolise, reproduce, and interact with their environment. Each bacterial generation would be, in some sense, "choosing" to continue the photosynthetic processes that were gradually transforming the atmosphere.
Process philosophy sounds like woo-woo nonsense to most people as we are so used to time-slicing things into discrete objects and moments. Further, accepting that consciousness is in “all things” is also a step too-far beyond knowing “what is like to be a bat” (Nagel). It’s just too woo-woo.
Ironically what gives Process philosophy credence in a too-rational world is the Simulation Hypothesis. That our current world is just one simulation instance and it’s entirely believable that the people and rocks and cyanobacteria are “conscious” processes running on some god-run substrate that gives then “intention”. (Religions might suggest their god is the super-operator of the simulation).
Stay with me….
De-anthropizing (is that even a word?) our AI companies motives and actions, we can say they are parts of a Whiteheadian process (within this world simulation 😀) and the SV tech bros have been given the choice for humanity to optimise for intelligence/computation (aerobic in our analogy) over other valuable exceptional (see “Human Exceptionism” section below) human traits (the anaerobic in our analogy) resulting in future guaranteed extinction of humanity. The SV tech bros make locally advantageous decisions that seem benign today but with extinction outcomes - the equivalent of cyanobacteria “choosing” photosynthesis.
The tech leaders aren't malevolent any more than cyanobacteria were - they're responding to immediate environmental pressures (competitive markets, venture funding, technological possibilities) by making moves that seem locally rational.
“Elevating” Consciousness
I’m not apologising for the tech bros, they aren’t mere puppets of their environmental circumstance, nor are we onlookers excused from passively accepting the circumstances whilst burning megawatts on cute images, to be forgotten within a few minutes.
No. I suspect that Eliezer’s call-to-action for all of us is to stop being NPCs or worse passively accepting the pro-ASI narrative and become activists for a rational challenge to whatever is presented as “safe” or “safe for now”.
The key here is a needed consciousness change:
“No problem can be solved from the same level of consciousness that created it.”
- someone (maybe Einstein)
Setting aside the simulation scenario and free-will debates, consciousness may be viewed to be just the accumulation of more information. Wrong, we know that the AIs have read the whole internet, so that is not a good frame. Another proposal might be due to embodied awareness - maybe. Another may be that a raising of consciousness a mysterious thing called Wisdom. For wisdom certainly is in short supply in the AI tech bro landscape and their advisors/elders are VC investors whose prime focus is money, money, money, power.
All Tech innovation including AI is completely beholden to investor’s goals, its an earlier alignment problem. How can we have AI alignment if the very creators of the AI aren’t aligned with humanity’s safety?
Back in 2023 I wrote about OpenAI’s non-profit structure, it was brave, it was good. The recent merging of OpenAI’s for-profit and non-profit structures is evidence that the moral foundations of the organisation are tenuous if not completely reduced to empty virtue signalling.
We need enough of society with a new elevated level of consciousness. The information is available, the human superpower is to acknowledge our mammalian addictions that are being hacked, overcome them and transform information into Wisdom. Then be embodied examples for other humans to model from - particularly our younger generations. From there, narratives about AI Alignment may shift to more proactive protection of humanity.
Human exceptionalism or what?
The cumulative effect of tech bro culture and our own tech addictions to our phones and “always on” services. AI’s footprint expands more voraciously than mobile phones and apps - all because it aids our laziness and instant gratification needs.
This is creating conditions incompatible with human flourishing or even survival.
So does humanity need to continue to exist? The joke amongst tech geeks is that carbon-based humanity is just the bootstrap program (BIOS) for the “betterer” silicon-based operating system.
Setting aside the Fermi paradox, we can confidently assert that humanity is worth preserving because being able to develop intelligence and culture to our current level is highly improbable and possible miraculous. I mean, seriously we are probably less than a century from Kardashev Level 1!
Human exceptionalism is a story we tell ourselves but can we be more specific? In 2025, nobody argues against the use of slide rules or calculators, they are a supercomputer for common math operations. But a belief that “intelligence” is the sole domain of humans we are clearly wrong as each benchmark is progressively clobbered by the largest AI models. Humanity is in retreat and because (in western culture) we have generally defined ourselves in terms of our jobs (“what do you do?”) we face a panic of purpose.
But there are other dimensions to humanity that may remain the sole domain or at least viable shared. Wisdom, empathy, prudence, craft, generosity, conviction, virtue may all be simulated by AI, BUT the uncanny valley will remain for those with just a little discernment. The joy of personal creation in the arts as opposed to just measuring by the outputs. The power of limits where people compete against their own inherent physical limitations. The skill and intuition of the protagonist in Iain M. Banks “Player of Games” hints at some faculties that are not replicable. Is there any machine analog to “spidey-sense”? (Robert Sapolsky would likely dismiss this outright).
I havn’t researched yet if there is a “Turing test” for other faculties but 4e cognition will be solely a unique superpower of humanity for some time.
Emad Mostaque offers a similar distinction in his recent book, “The Last Economy”.
AI is the Action Layer. It is the uncapped, infinitely scalable engine of execution and optimization. It can achieve any well defined goal with terrifying, inhuman efficiency.
Humanity is the Alignment Layer. We are the source of the values, the ethics, the preferences, and the ultimate purpose that guides the machine's optimization. The "Arts of Being Human," our capacity for wisdom, taste, moral judgment, and love, are no longer "soft skills." They are the most crucial economic input in the entire system. We are the compass for the rocket ship. But this cannot be a passive role. We cannot simply wish for better values. We must engineer the channels through which these values are transmitted.
and
Computation vs. Consciousness: The fundamental distinction between AI and human intelligence. Computation is the syntactic engine for answering "how." Consciousness is the semantic engine for answering "why." The ultimate Human- AI Symbiosis is the fusion of these two (see Chapter 25).
Call to Action
I recommend reading these books rather than getting the Youtube influencer view. Afterall “the machine” and its algorithm will ensure you don’t get to learn about these robust viewpoints and you will only be fed the Altman Alt-reality.
++ I recently re-watched the 1960 Time Machine movie of H.G Wells book had some prescient message for an existence without work. It’s definitely worth a look! Side note: Aussie Rod Taylor was the star, in the 2002 re-make Aussie Guy Pearce was the protagonist. Spookie.
** yes I know about Eliezer’s HP fan fiction and I suspect this current book dramatizes the main extinction scenario, but this really is Paul’s day job.