Superintelligence is a hypothetical AI that far exceeds the best human minds in virtually every field, from science and engineering to strategy and persuasion. It is usually described as the step beyond AGI, and it is the subject of both the boldest promises and the gravest warnings in the debate about AI.
Where the idea comes from
In 1965, the statistician I. J. Good, who had worked with Alan Turing as a codebreaker, imagined an ultraintelligent machine that could design even better machines. The result, he wrote, would be an “intelligence explosion” – the idea behind Recursive Self-Improvement, FOOM and the Singularity. Good added a condition: such a machine would be the last invention humans need only if it were docile enough to tell them how to keep it under control. The philosopher Nick Bostrom made the term widely known with his book Superintelligence (2014). He distinguished systems that are much faster than humans, systems made of many cooperating minds and systems that are simply smarter, and argued that keeping such a system aligned and controllable would be the decisive problem.
From thought experiment to business plan
In the 2020s, superintelligence moved from philosophy seminars into company names. Ilya Sutskever, a co-founder of OpenAI, co-founded Safe Superintelligence Inc. in June 2024, saying its only product would be a safe superintelligence. In June 2025, Meta created Meta Superintelligence Labs. In July 2025, Mark Zuckerberg set the goal of “personal superintelligence” for everyone; in August 2026, he expanded it into the essay The Future is for Everyone, which we analyzed as a case for distributing such power rather than concentrating it. Supporters expect superintelligent systems to accelerate science, medicine and growth – the hope behind ideas such as a country of geniuses in a datacenter and longevity escape velocity.
The case for caution
Critics fear that a system far smarter than its makers could pursue goals they never intended and could not stop – the worry captured by the Paperclip Maximizer, the Orthogonality Thesis and containment. In October 2025, the Future of Life Institute published a short statement, signed by the AI pioneers Geoffrey Hinton and Yoshua Bengio among many others. It calls for a prohibition on developing superintelligence until there is broad scientific consensus that it will be done safely and controllably, and strong public buy-in. In September 2026, Senator Bernie Sanders and Representative Greg Casar introduced the Ban Artificial Superintelligence Act. It would pause the largest models until a new federal department sets rules, and ban superintelligence outright. After the midterms, the House co-sponsor Yassamin Ansari called for it to top the Democratic agenda on day one.
Others doubt that superintelligence is near, or even coherent as a single concept. The computer scientists Arvind Narayanan and Sayash Kapoor argue that intelligence is not one quantity that can simply be scaled up, and that AI is better understood as a normal, if powerful, technology.
Why it matters
Whether or not superintelligence arrives soon, the word shapes investment, regulation and public fear today. Supporters of a ban argue that no one should build what no one can yet control. Opponents, aiciety among them, argue that a ban would bind careful developers while reckless ones carried on, and favor open testing, oversight and law instead. Either way, the most useful questions are concrete: who would control such systems, how their goals would be checked and how their benefits would be shared.
Written with Claude, made by Anthropic, whose chief executive coined the phrase “country of geniuses in a datacenter.”
Related
AGI · Alignment · Singularity · FOOM · Recursive Self-Improvement · Containment · AI Pause · p(doom) · Frontier Model
Sources: I. J. Good: Speculations Concerning the First Ultraintelligent Machine (Advances in Computers 6) (1965) · Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford University Press, 2014) · Safe Superintelligence Inc. (June 19, 2024) · Mark Zuckerberg: Personal Superintelligence (July 30, 2025) · Mark Zuckerberg: The Future is for Everyone (August 10, 2026) · Future of Life Institute: Statement on Superintelligence (October 22, 2025) · The Guardian: Harry and Meghan join AI pioneers in call for ban on superintelligent systems (October 22, 2025) · Ban Artificial Superintelligence Act, H.R. 10538 (GovInfo) (September 24, 2026) and S. 5493 (September 23, 2026) · Arvind Narayanan and Sayash Kapoor: AI as Normal Technology (Knight First Amendment Institute) (April 15, 2025)
Leave a Reply