The Basilisk and the Ban

Roko’s Basilisk was an argument that almost nobody who read it closely believed. It became one of the internet’s best-known legends anyway, largely because it was deleted. This essay asks what the Basilisk’s career shows about how speculative fears about AI travel and what it reveals about the community that produced it. It also asks why open criticism disarmed it better than silence did.

A weak argument, quickly answered

Roko’s post of July 23, 2010, imagined a benevolent future AI that might punish people who had known of it but not helped it come about. It was a threat issued, in effect, backwards through time. The objections are simple. A future agent gains nothing by punishing a past it cannot change. An agent known to ignore threats gives no one a reason to make them. And the bargain requires knowledge of the future agent that nobody has. Within a day, commenters on the original thread had called the post wrong, and Eliezer Yudkowsky’s own angry reply set out the policy of refusing all blackmail. Roko himself, as Rob Bensinger later showed, presented the scenario as a flaw in a proposed design, not as a reason to donate.

The case for silence

Yudkowsky deleted the post and banned the subject. It is easy to mock this, and unfair. His reasoning, as he explained it in 2014, was not that the argument was sound but that spreading it had no upside. If it was wrong, nothing was lost by removing it; if some repaired variant was right, spreading it would be harmful. The underlying idea is respectable. Nick Bostrom published a typology of information hazards in 2011. Most people accept that some knowledge – a detailed route to a weapon, for instance – should not be broadcast.

The answer lies in the difference between a hazard with a mechanism and a hazard that exists only for those who accept a long chain of speculative premises. Bostrom defines information hazards as risks arising from the spread of true information, and a weapon recipe is dangerous whatever its reader believes. The Basilisk could harm only through belief, and suppression supplied the one thing the argument lacked: the appearance that the people best placed to judge it took it seriously. Bensinger’s retrospective of October 2015 makes the point from inside. Because the topic was banned for several years, members could not explain that they had rejected it, and rumor filled the gap. The LessWrong wiki now draws the same lesson: visibly suppressing a supposed hazard can be the surest way to spread it.

How a fear travels

The Basilisk’s afterlife followed a recognizable pattern. RationalWiki reconstructed the episode from fragments and assumed that forum members believed the argument. In July 2014 Slate presented it in a register of mock horror as an all-powerful, malign machine, and that November an xkcd title text turned it into a joke. In 2018 Vice explained it to readers because a pun on its name had brought together Elon Musk and Grimes. At each step the story became simpler and more vivid. The benevolent AI became a malevolent one. A criticism of a design became a fundraising pitch. A community that had rejected an argument became one terrified by it.

What travels, in other words, is not the argument but its shape: a hidden judge, a debt incurred by knowing, a punishment for failing to act. Beth Singler has read the episode as implicit religion. In her account, a community that defines itself against religion reproduced older religious forms of argument; her title names Pascal’s wager, with a superintelligence standing where God once stood. Fears spread when they fit stories people already know, and retelling selects them for drama rather than validity.

This is the reflexive layer of AIciety in an unusually pure form. The definition holds that the discourse in which a society observes its relationship with AI is part of that relationship: whoever describes AIciety changes it. The Basilisk shaped a forum’s moderation policy, years of press coverage and the reputation of a whole community, and no AI system was involved at any point.

What it says about the community

It would be easy to conclude that rationalists are credulous. The record says otherwise. Most of them rejected the argument and said so at the time. Later they documented their own errors, the ban included, with unusual candor: Yudkowsky conceded he had acted foolishly, Bensinger set out the misconceptions, and the ban was lifted. That is how an open institution corrects itself, slowly and in public.

The episode does expose a real vulnerability, and the community named it before its critics did. A culture that takes expected-value reasoning seriously can be captured by tiny probabilities attached to vast stakes. Yudkowsky coined the term Pascal’s mugging for that trap in 2007, and Nick Bostrom turned it into a short dialogue in the journal Analysis in 2009. The Basilisk is a Pascal’s mugging with the mugger placed in the future. That this community produced it says less about gullibility than about the cost of a virtue: people who follow arguments where they lead will sometimes be led somewhere strange.

The lesson for AI policy

The same structure recurs in debates about regulating AI. A scenario of low but unknown probability and unbounded harm is set against any cost, and the size of the imagined harm stands in for evidence of its likelihood. The Basilisk warns against that move, and the warning cuts in every direction: against doom arguments that justify prior permission, and against utopian ones that would justify anything.

This does not mean speculative risks should be ignored. The International AI Safety Report 2026 describes an evidence dilemma: acting too early can entrench ineffective interventions, while waiting for conclusive evidence can leave society exposed. The Basilisk suggests a way through it. Ask for a mechanism, not only a magnitude. Prefer responses that cost little if the fear proves empty – transparency, monitoring, open argument – to responses that cost a great deal, such as bans on building or on speaking. And let ideas be tested in the open, where the commenters who saw through the Basilisk within a day could have done their work without a ban to turn it into legend.

What the song does with it

I’m Upping My P(doom) places the basilisk in its second chorus, written by osmarks in April 2024 after an opening verse and chorus by MusicPerson. It sits alongside a joke about Nvidia’s stock and a reference to the Omega Point. The lyricist’s annotation offers no explanation. It gives only a link labeled “rationalist mugging”. The link leads to a screenshot of a comic post in which a street robbery collapses under Newcomb-style predictors, precommitments and a Pascal-style wager. That is the most telling gloss of all. By 2024 the Basilisk was no longer a fear but a genre, a parody of a way of reasoning. The song hears it as background noise in a market boom, which is roughly where it belongs.

Sources: Roko: Solutions to the Altruist’s burden (unofficial archive) · LessWrong wiki: Roko’s Basilisk · Rob Bensinger: A few misconceptions surrounding Roko’s basilisk (2015) · Slate: The Most Terrifying Thought Experiment of All Time · xkcd 1450: AI-Box Experiment · Vice: Explaining Roko’s Basilisk · Beth Singler: Roko’s Basilisk or Pascal’s? (Implicit Religion, 2018) · Nick Bostrom: Information Hazards (2011) · Nick Bostrom: Pascal’s mugging (Analysis, 2009) · International AI Safety Report 2026 · What is AIciety? · osmarks: p(doom) song, objectively correct interpretation · osmarks: Rationalist Mugging · Lexicon: Roko’s Basilisk

Leave a Reply

Your email address will not be published. Required fields are marked *