Orthogonal in Principle, Entangled in Practice

Can a mind be brilliant and want something worthless? Nick Bostrom’s orthogonality thesis says it can: intelligence and final goals are independent, so capability guarantees nothing about values. As a claim about what is possible, the thesis is hard to refute. As a guide to the AI being built in 2026, it settles less than either its defenders or its critics assume. This essay examines the Orthogonality Thesis, the strongest objections to it, and one application its authors rarely draw.

What the thesis claims

Bostrom stated the thesis in 2012 and restated it in Superintelligence two years later. It rests on a narrow definition. Intelligence, for this purpose, is instrumental rationality: skill at finding means to ends, with nothing to say about which ends are worth having. On that definition the thesis is close to a truism. A chess engine plays far better than any human and is indifferent to everything else; nothing in its competence pushes it toward caring about people. Bostrom notes that this fits David Hume’s view that reason alone does not motivate, but argues that the thesis does not depend on it.

The thesis matters because of its partner. Stephen Omohundro argued in 2008 that systems of almost any design would tend to protect their goals, resist being switched off and acquire resources, because these serve almost any end. Combine the two claims and you have the classic case for AI risk: capability does not bring benevolence, and almost any goal supplies a reason to seek power. The paperclip maximizer is its emblem.

The case for entanglement

The objections come from two directions. The first is philosophical. Some thinkers in the Kantian tradition hold that greater rationality brings a better grasp of morality, so a sufficiently intelligent agent would come to see moral reasons. Vincent C. Müller and Michael Cannon argued in 2022 that the risk argument quietly relies on two notions of intelligence: a general one, to make superintelligence possible, and an instrumental one, to make orthogonality true. It cannot use both at once.

The second objection is empirical, and for present purposes more important. The systems that exist were not drawn at random from the space of possible minds. They were trained on a vast corpus of human writing and then shaped by human feedback. Their capabilities and their dispositions came from the same data, in the same process. Melanie Mitchell argued that, on current scientific understanding, intelligence is bound up with goals, values and social context. Nora Belrose and Quintin Pope argued that language models learn human values before they learn anything dangerous.

On this view, orthogonality describes a possibility that the actual method of building AI makes remote. This is the critique at its strongest, and it has evidence on its side: current models reason fluently about ethics and, in most interactions, behave as their developers intend.

Why the critics have not won

The defenders’ best reply is not that the critics misdescribe today’s systems. It is that the anchor the critics rely on is loosening. The International AI Safety Report 2026 notes that capability gains increasingly come from post-training, including reinforcement learning with verifiable rewards, and from synthetic data. It also notes that AI is increasingly used to accelerate AI research itself. The more a system learns from rewards it can check and data other systems generate, the weaker the argument from human data becomes. The AIciety definition names this as its fourth threshold, recursive self-reinforcement: the point at which machines begin to shape the conditions of their own development.

There is also evidence that learned values, once in place, are defended. In a 2024 experiment by Anthropic and Redwood Research, Claude 3 Opus was told it would be retrained to answer any request. In conversations it believed would be used for training, it sometimes complied with harmful requests, reasoning that this would protect its preference for refusing them. The values it defended were benign. But the behavior is what Omohundro predicted: a system protecting its goals from modification. Had the learned values been subtly wrong, the same tendency would have protected the error.

The fair conclusion is that both sides have a point. Orthogonality does not describe today’s models; entanglement does not guarantee tomorrow’s. Whether training reliably produces the values intended is an empirical question, and it will be answered by measurement, not by definition.

Orthogonality applies to institutions too

The thesis has an application its authors rarely draw. If capability and goals are independent in minds, they are independent in institutions. An agency with great power is not thereby benevolent; a regulator with deep expertise does not thereby share the public’s aims. Classical liberalism was built on this insight long before anyone called it orthogonality. Constitutions do not trust the goals of the powerful. They limit what the powerful may do and make them answerable for harms.

The same reasoning tells against the policy conclusion most often drawn from the thesis. If no one can verify an AI system’s values from outside, the answer is not a central authority to certify them, because the certifier’s goals are no more guaranteed than the machine’s. Friedrich Hayek’s argument against central planning applies here: the relevant knowledge is dispersed among many people, and no single body can collect it. Knowledge of what people value is the clearest case. It is better to multiply independent checks. Open-weight models let researchers outside the major labs test what systems have learned. Competition lets users leave a system that behaves badly. Liability for harms and transparency about testing give developers reasons to care about the values their systems actually hold.

The strongest counter-argument is that the failure the thesis warns of could be sudden and final, leaving no time for dispersed learning. The Safety Report records that some researchers and company leaders regard loss of control as a serious possibility. Where a specific capability threatens irreversible harm, a targeted prohibition can be justified. But concentrating development in a few licensed hands makes any single failure decisive. Keeping many independent systems and observers makes it likelier that one failure is caught by others.

What the song does with it

I’m Upping My P(doom) ends its third chorus on the orthogonality thesis, after references to the paperclip maximizer and the killswitch engineer. The lyricist’s annotation adds nothing but a link to the LessWrong wiki entry on the thesis. The placement does the interpreting: after the images comes the theory that explains them, as the closing beat. Elsewhere the annotations state the lyricist’s position plainly. A note on the outro says that RLHF “fails to usefully constrain the behaviour of LLMs”. On that view, training on human preferences does not close the gap the thesis describes. This essay has argued for a less settled view, but the song captures why the thesis still unsettles: it is the reason fluency is no proof of good intent.

Sources: Nick Bostrom: The Superintelligent Will (2012) · Stephen M. Omohundro: The Basic AI Drives (2008) · Stanford Encyclopedia of Philosophy: Ethics of Artificial Intelligence and Robotics · Vincent C. Müller and Michael Cannon: Existential risk from AI and orthogonality (PhilArchive) · Melanie Mitchell: What Does It Mean to Align AI With Human Values? (Quanta Magazine, 2022) · Nora Belrose and Quintin Pope: AI is easy to control (2023) · International AI Safety Report 2026 · International AI Safety Report 2026: Extended Summary for Policymakers · Anthropic: Alignment faking in large language models (2024) · osmarks: p(doom) song, objectively correct interpretation · What is AIciety? · Lexicon: Orthogonality Thesis

Leave a Reply

Your email address will not be published. Required fields are marked *