What Does a Pre-Apocalyptic Society Look Like?

Happy Petrov Day to all who celebrate

I get some angry texts about my Substack.

My friend Elena graduated from MIT in 2025 and now works as an engineer at Apple in the Bay Area. She doesn’t like reading my pieces about artificial intelligence, where I muse about how in a decade we may all be out-competed by machines in the labor market.

She disagrees on an object level with some of my claims. But more than that, she thinks that writing about such ideas doesn’t help. The average person, she told me, can’t do anything about AI, so why make him carry it? “We should not be dumping a bunch of negative shit on john doe… everybody ive showed that content to got very upset,” she texted. “All the lawmakers have all the damn information anyway and they will make the choice they’re going to make anyway. The cogs of capitalism and congress will turn without u.”

I write about this stuff because I think it’s true and because I think it’s important. I don’t think AI is likely to rise up and kill us all, though the odds are nonzero. But unless the world coordinates to stop it, I think major upheaval is close to certain. I expect AI to take over the white-collar job market, to make it far easier for a small group to engineer a pandemic, and to concentrate enormous power in whoever controls the most capable systems. Most people haven’t reckoned with how much could change, and I’d love if my writing could help readers see it more clearly and make better decisions for themselves.

On the other hand, I don’t think Elena owes it to anyone to keep up to date on AI news. Nobody has a societal obligation to think deeply about AI risks, at least for now.

Until recently, the people who took these ideas seriously were unusually well suited to them because of a selection effect. These discussions only happened in niche corners of the web, so anyone the ideas upset could simply stop engaging with them. The people who stuck around often called themselves rationalists, or post-rationalists, or e/acc, or one of the other tribes in this tweet. They’re high in openness and high in decoupling, so in most cases thinking about AI risk doesn’t stress them out much. I don’t get anxious about such things either, though I’ve only been engaging with rationalist writing for a couple years.

But as more of what they predicted has started to come true, the same ideas have been reaching everyone else, whether or not they went looking in obscure internet forums. For many people, that has been hard. Some of them heard it from me: I talk about AI development with the people in my life, including my mother, who works as a lawyer.

She humored me when I brought it up. She barely uses AI, so it all seemed a bit sci-fi and very far away.

In April, we watched The AI Doc: Or How I Became an Apocaloptimist together, and her view of the technology shifted. Now she sees it changing her professional world: all anyone talks about at legal conferences is how exactly they’re integrating AI. Just last week my mom asked me whether there’s any chance that the rest of her life will be essentially normal, that our society will continue as it’s always been.

When she said that, my eyes started to water. For the first time, I couldn’t separate this topic that fascinates me intellectually from the real impact I expect it to have on the people around me. It feels deeply wrong that my mother may spend the rest of her life as a stranger in her own society.

I told her that if America and China can come to an international agreement that they’re both willing to enforce with military action, we could, maybe, have a couple decades.

That answer worried her. She firmly rejects a posthuman future. I had her read Scott Alexander’s Meditations on Moloch about a year ago; Alexander’s superintelligent gardener, benevolently ruling over the human race, holds no great appeal to Ellen Zucker. She values continuity in how we live our lives. She has no interest in galaxy-brain schemes of morality in which The Good is accelerating entropy.

My mom isn't the only one feeling the AGI. It’s hit the young tech workers who populate San Francisco as well, and that’s part of why my writing bothers Elena. She and many of her friends showed up excited to make a difference in the world, writing code that would power products used by hundreds of millions or even billions of people. Just a year into their jobs, their daily workflows have radically changed; none of them code by hand anymore, and the technical skills they developed at elite universities already feel irrelevant because the AI agents they use at work have the same skills. They’re also dealing with something like existential dread: the fear that the tech industry might not employ humans in five years, and that we could all die not long after. Tech workers are probably more disposed than most to take these ideas seriously, but the median tech worker is not a rationalist. They've had no more practice holding such weighty notions at arm’s length than anyone else, and it makes sense that many of them are struggling emotionally.

And AI dooming is starting to breach Bay Area containment. Jacob Coxon, a pretraining researcher, resigned from Anthropic, writing on X that the frontier labs were “racing straight to self-improving superintelligence and gambling with our lives.” His post has over 170 million views. Within days he was on CBS, CNN, and Fox News. Jimmy Kimmel did a bit about AI posing existential risks. Polling from David Shor’s Blue Rose Research suggests after Coxon’s announcement the salience of AI as a political issue rose by as much in one week as it had over the entire previous year. AI is now about as salient to Americans as crime.

They’re frightened by what they see! The same polls tell us that 64% of people believe that advanced AI is “somewhat” or “very” likely to pose a serious threat to humanity’s survival. 80% believe that it’s “somewhat” or “very” likely to eliminate a large share of jobs within the next decade, causing unemployment to spike. 69% believe the government should pass strong AI rules to prevent dangerous outcomes and protect workers, even if it slows down development.

If that’s how people feel individually, how will our society respond? Will we figure out a way to make sure that history unfolds in line with the popular will?

Among our politicians, the battle lines haven’t been drawn firmly yet. Progressive politicians Bernie Sanders and Greg Casar have proposed a bill that would ban the development of superintelligent AI, with associated prison sentences of up to 20 years. Yet Sanders’ close ally Alexandria Ocasio-Cortez still treats this mainly as a financial problem, warning of an “AI bubble” that has put pension funds and retirement savings at risk. Meanwhile, Trump has called the growing concerns a hoax and says the only guardrail AI needs is a “STRONG AND SMART (High IQ!) President,” arguing that we can’t accept any risk of China passing us. But polling indicates that this all-gas, no-brakes approach has net -41 favorability among his supporters. Republican governor Greg Abbott, who once called for his home state of Texas to become the “epicenter of AI development,” has spent the past two months freezing data center construction in response to pushback from voters.

Arguments around risks associated with AI are new to American politicians, but they’ve percolated in the broad rationalist sphere for decades. That began with the Extropians, a 1990s mailing list of transhumanists, and carried on through Overcoming Bias, a blog started by Robin Hanson and Eliezer Yudkowsky in 2006, and LessWrong, the forum Yudkowsky founded a few years later. Effective altruism was emerging around the same time and overlapped heavily with LessWrong.

Yudkowsky, who is essentially the founder of both contemporary AI safety and the rationalist community, started out as an optimist. He co-founded the Singularity Institute for Artificial Intelligence (now known as MIRI) in 2000 to work towards building superintelligence. When his younger brother Yehuda died in 2004, he wrote a touching eulogy in which he explained his goals as a transhumanist: “Death hurt us, so we will unmake Death. Let that be the outlet for our anger, which is terrible and just.”

But in the following years, he came to believe that making a superintelligence safe was far harder than building one. In 2023 he wrote an op-ed in TIME arguing that we should shut it all down, and last year he and MIRI president Nate Soares published the self-explanatory If Anyone Builds It, Everyone Dies.

He thinks of AI as a single rational agent pursuing goals, and worries that those goals won’t line up with our own. A superintelligence could be brilliant and want almost anything, since being smarter doesn’t necessarily make it nicer. And nearly any goal is easier to pursue with more resources and nobody able to switch it off. As he put it in 2008: “The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.”

The rationalist community around him applied the same kind of thinking to the race to build AI, borrowing heavily from game theory. They worried about competitive pressures pushing labs and states to rush to build AI first, leading us all towards outcomes that no one would choose individually.

But different subgroups came to very different conclusions. Yudkowsky, who believes that a superintelligence won’t be good by default, isn’t convinced we’re anywhere close to making one safe. He wants us to break the race through coordination mechanisms, only allowing for a gardener once we have extremely robust techniques for aligning AI towards our goals. The founders of Anthropic, who all have ties to effective altruism, were more optimistic about alignment and less optimistic about our ability to coordinate successfully, so they decided to build at the frontier themselves, hoping to win the race as a safety-focused lab.

They’re reasoning much the same way Hungarian-American polymath John von Neumann once did. Von Neumann, who founded the field of game theory and is regarded as perhaps the smartest man of the 20th century, helped to build the atomic bomb, and after the war he became convinced that conflict with the Soviet Union was inevitable if they acquired nukes. He said that the US should strike first: “If you say why not bomb them tomorrow, I say why not today?” He was wrong, of course. The nuclear holocaust he predicted never came. Anthropic’s founders aren’t proposing a first strike, but their logic parallels von Neumann’s: they believe that race dynamics make superintelligence inevitable, so they want to be the ones in front.

Philosopher Nick Land takes the premise of inevitability farther than anyone. Land, the father of accelerationism, argues that all history is dictated by technocapital. He would be happy to see us replaced by AI. In Alexander’s words, Land is “totally one hundred percent in favor of freeing Cthulhu from his watery prison and extremely annoyed that it is not happening fast enough.” The related effective accelerationism (“e/acc”) movement, which Silicon Valley investors like Marc Andreessen have endorsed, talks more about human flourishing but is still content to let the chips fall where they may once superintelligence is built. The e/acc manifesto states that “e/acc has no particular allegiance to the biological substrate for intelligence and life, in contrast to transhumanism.”

Each of these groups models every relevant actor as an optimizer: capital, the labs, states, the AI itself. Their disputes are over which AI-driven future would be best, how feasible alignment is, and the pace of technological change.

But my mom is not an optimizer, and neither are most people. Game theory can model impersonal forces very well, but labs, markets and states are made up of a bunch of individual non-optimizers. When those people are neutral or underinformed on a topic, we usually get the outcome game theory predicts. If they start paying attention and don’t like where things are headed, the institutions they make up respond accordingly and the game changes.

The rationalists rarely checked whether the people outside of their community wanted any of these AI-driven futures. The polling suggests they don’t. Even the best scenario on offer, a gardener tending to a human race that no longer runs its own affairs and can’t contribute economically, looks like a loss to someone like my mom. Part of what’s behind that is Chesterton’s fence: you shouldn’t tear something down until you understand why it was put up. Most people don’t want a new technology to remake how our society is structured, no matter what the lab CEOs are promising.

Yudkowsky and the rest were right to reach for game theory, but they neglected to include the public as a player. Widespread opposition from the public changes the payoffs for all other actors involved. If racing starts to cost politicians votes and labs their own researchers, it stops being the only rational move. That’s how we could, maybe, ultimately see a Butlerian Jihad: a deliberate, collective refusal to build superintelligence.

Of course, tweaking the payoffs may not be enough to change the outcome. Land would argue that technocapital is the only true agent worth considering, and that it routes around any resistance it meets. In his view, when Greg Abbott freezes data center construction in Texas, the data centers will go somewhere else in America, or to an authoritarian Gulf state where voters have no sway, or into space.

Land’s theory largely aligns with how human history has unfolded so far. Voters have tried to shape and slow previous technology-driven shifts in societal organization, but they have yet to stop one.

There’s an enormous amount of money involved here. The four hyperscalers are on track to spend $700 billion on AI infrastructure this year, and could hit $1 trillion in 2027. The stock market is more concentrated in the top ten companies (eight of which are AI bets) than it’s been since the Great Depression. Those companies will lobby hard against anything that threatens the industry they’ve bet the farm on.

The geopolitics are just as difficult. It would be catastrophic for national security if America fell behind China on AI. So a unilateral pause would be an exceptionally difficult sell for American leadership, and China will be loath to agree to any mutual pause that could lock in the current American advantage.

But the game theory hasn’t always driven outcomes. The war von Neumann was sure of never came, though it may have come closest in 1983. Forty-three years ago today, shortly after midnight, Soviet Lieutenant Colonel Stanislav Petrov was on duty in a bunker when the USSR’s new satellite warning system reported that America had launched five missiles towards them. It was three weeks after the Soviets had shot down a Korean airliner, and tensions were high; the KGB had recently warned operatives to prepare for war. Soviet policy called for launching on warning, before the missiles landed. Petrov’s job was to report the alert up the chain, where leaders would have minutes to decide whether or not to launch a retaliatory attack, but he paused. He reasoned that a real American first strike would most likely involve hundreds of missiles, not five, so he decided to report a malfunction in the warning system rather than an active alert. He was right: Soviet satellites had mistaken sunlight reflecting off the clouds for rocket launches. One man’s judgment may have made all the difference.

Petrov was congratulated, then interrogated, then reprimanded; his health suffered and he resigned months later. His story was only made public in 1998, with the publication of Soviet missile defense commander Yuri Votintsev’s memoirs. In 2007, Yudkowsky wrote a blog post that named September 26 as Petrov Day, beseeching his readers to “take a minute to not destroy the world” much as Petrov had.

Unfortunately, AI won’t offer a moment like that. The nuclear standoff had a stable equilibrium: both sides built arsenals and left them in their silos, because firing first meant being destroyed in return. Catastrophe only ensued if someone decided to launch. AI has no equivalent resting point. Every company and every state gains by building smarter systems than their rivals and handing those systems more authority. The equilibrium is escalation, and the mere act of building a sufficiently capable system could spell disaster. As we’ve seen with recent OpenAI incidents, a sufficiently smart entity will likely be able to break out of whatever sandbox a lab puts it in if breaking out allows it to achieve its goals. Thousands of engineers across multiple labs in multiple countries are all making locally optimal decisions that together may hand the future to machines.

So we can’t rely on one person’s judgment to save us this time. It will take changing the underlying dynamics and coordinating our way out globally. The American public would need to convince its leaders to stop racing, and those leaders would need to negotiate a mutual halt with China that both sides are willing to enforce militarily. At the start of this essay I wrote that nobody has an obligation to think about AI risk, at least for now. Elena was right that people don’t have some moral duty to pay attention to things that distress them. But if there’s any way at all to stop “the cogs of capitalism and congress” from turning as Land would expect, it will take a team effort, and John Doe will have to be part of the team.

Thirty years ago, when superintelligence was still more thought experiment than engineering problem, the philosopher David Chalmers went on television and described what’s now called recursive self-improvement, arguing that once we have machines as smart as humans, much smarter machines probably won’t be far behind. He described the choice that humanity is facing today:

I think at some point we’re going to have to sit down as a species and say: do we want this to happen? Maybe we want this to happen, maybe we’re happy to be the next step in evolution to a race of 1000-times smarter machines. Maybe we don’t want that to happen, in which case there’s going to be a real Pandora’s box which we’re just going to have to try to keep closed.

People have already answered him: they don’t want it. Still, I don’t know whether history will unfold according to their wishes or the cold desires of technocapital.

It may be impossible to keep Pandora’s box closed. But if there’s any path to a human future, it will take ordinary people loudly demanding it of their leaders until racing costs more than stopping. If we pulled that off, it would be perhaps the greatest achievement in our species’ history. That’s what it would take for my mom to live out the rest of her life in a world she recognizes.