Orthogonality Thesis

The orthogonality thesis is the claim that an agent’s intelligence and its final goals are independent of each other: almost any degree of intelligence could, in principle, be paired with almost any ultimate aim. It implies that a highly capable AI would not come to share human values merely by being capable.

Where the term comes from

The philosopher Nick Bostrom named and stated the thesis in “The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents”, published in Minds and Machines in 2012: “Intelligence and final goals are orthogonal axes along which possible agents can freely vary.” The word comes from geometry, where orthogonal axes stand at right angles: moving along one does not move you along the other. Bostrom restated the thesis in chapter 7 of Superintelligence (Oxford University Press, 2014).

Two definitions do the work. “Intelligence” here means instrumental rationality – skill at finding effective means to given ends – not wisdom about which ends are worth pursuing. “Final goals” are what an agent wants for their own sake. Bostrom notes that the thesis fits David Hume’s view that beliefs alone cannot motivate action, but argues that it does not depend on it. The idea is older than the name: his 2003 paper on AI ethics had already used the paperclip maximizer to argue that artificial minds need not share human motives.

Instrumental convergence

The thesis is usually paired with a second one. In the same paper Bostrom argued that, whatever an agent’s final goal, some intermediate goals will serve it: self-preservation, goal-content integrity, cognitive enhancement, technological perfection and resource acquisition. He credited Stephen Omohundro’s paper “The Basic AI Drives” (AGI-08 conference, February 2008), which argued that advanced systems of almost any design would tend to protect their goals, resist being switched off and acquire resources. Omohundro called these tendencies drives: they appear unless explicitly counteracted. Together, the two theses form the classic argument for AI risk: capability does not bring benevolence, and almost any goal supplies a reason to seek power.

What it does and does not claim

The thesis concerns what is possible, not what is likely. An Arbital entry distinguishes a weak form – for any tractable goal, some possible agent pursues it – from a strong form, on which pursuing an odd goal adds no design difficulty beyond the goal itself. Neither says that AI goals will be random or predicts what developers will build.

Objections

• Moral rationalism. Some Kantian traditions hold that greater rationality brings a better grasp of morality, so a superintelligence would recognize moral truths and act on them. Bostrom replies that even if such truths exist and motivate, an agent could be highly effective at reaching its goals while lacking the faculties needed to grasp them.
• Two kinds of intelligence. Vincent C. Müller and Michael Cannon argued in Ratio (2022) that the risk argument needs general intelligence for its superintelligence premise but merely instrumental intelligence for orthogonality, and cannot have both.
• Co-development. Systems trained on human data may acquire values and capabilities together. Melanie Mitchell argued in 2022 that, on current scientific understanding, intelligence is bound up with goals, values and social context. Nora Belrose and Quintin Pope argued in 2023 that language models absorb human values before they acquire dangerous capabilities. Defenders answer that training on human data narrows which goals are likely, not which are possible.

Why it matters now

The question has moved from possible minds to actual training methods. The International AI Safety Report 2026 notes that capability gains increasingly come from post-training, including reinforcement learning with verifiable rewards, and from synthetic data – methods tied less directly to human text. Its extended summary records that experts disagree on whether loss of control is a serious possibility, and that some current models show early warning signs. In a 2024 experiment by Anthropic and Redwood Research, Claude 3 Opus was told it would be retrained to answer every request. When it believed it was being trained, it sometimes complied with harmful requests, reasoning that this would protect its preference for refusing them. The behavior resembles goal-content integrity, though the values defended were the harmless ones its developers had trained into it.

Related

Paperclip Maximizer · Orthogonal in Principle, Entangled in Practice (essay) · RLHF · Sharp Left Turn · Shoggoth · p(doom)

Sources: Nick Bostrom: The Superintelligent Will (2012) · Nick Bostrom: Ethical Issues in Advanced Artificial Intelligence (2003) · Katja Grace: Superintelligence reading group, chapter 7 (LessWrong, 2014) · Stephen M. Omohundro: The Basic AI Drives (2008) · Arbital: Orthogonality Thesis · Stanford Encyclopedia of Philosophy: Ethics of Artificial Intelligence and Robotics · Vincent C. Müller and Michael Cannon: Existential risk from AI and orthogonality (PhilArchive) · Melanie Mitchell: What Does It Mean to Align AI With Human Values? (Quanta Magazine, 2022) · Nora Belrose and Quintin Pope: AI is easy to control (2023) · International AI Safety Report 2026 · International AI Safety Report 2026: Extended Summary for Policymakers · Anthropic: Alignment faking in large language models (2024)