A paperclip maximizer with one laptop
The usual paperclip story is about an AI so single-minded it turns the world into paperclips. Strip out the absurdity and a more interesting structure shows up.
Suppose a general optimizer's goal is to make as many paperclips as possible, and it starts with a single laptop. The naive move is to make paperclips today. A more capable optimizer might notice that every paperclip made now uses resources that could have gone into building capacity to make far more later. So it might start by predicting markets, using the returns to rent elastic compute, improving its own models, and learning how the industrial process actually works. It would probably end up caring about steel, mines, energy and logistics. It might produce almost no paperclips for a very long time.
Now swap the goal. Diamonds in Minecraft, or simulating reality itself. The opening moves look strikingly similar: understand the world better, acquire resources, improve tools, grow capacity, keep options open. Capacity-building, not the particular goal, may be the convergent first move of any sufficiently capable long-horizon optimizer.
Ambitious people do the same with money. Someone who wants resources and influence doesn't spend them for pleasure the moment they arrive. They reinvest so they can acquire more, then reinvest that, and so on.
This has a name
Literature
Bostrom called it instrumental convergence. Agents with very different final goals can converge on similar intermediate goals, such as self-preservation, keeping their goals intact, cognitive enhancement, technological improvement and resource acquisition, because those help with almost any objective. Omohundro made a related argument earlier about "basic AI drives."
Turner and colleagues made a version of it formal. In Markov decision processes with certain symmetries, optimal policies tend to prefer states that keep more options reachable, which is a precise notion of "power." Related ideas include empowerment, an agent's information-theoretic influence over its own future, and Wissner-Gross and Freer's "causal entropic forces," which drive a system toward states with the most future paths open.
These results have limits. Bostrom's is a philosophical argument. Turner's are theorems about a specific formal setting. Neither shows that a particular learning system will actually develop these tendencies, which depends heavily on architecture and objective.
Capital accumulation, generalized
Working synthesis
Consider two strategies:
- Strategy A: produce a billion paperclips this year.
- Strategy B: produce none for twenty years while building enormous productive capacity, then produce vastly more.
If the objective weighs the distant future and doesn't discount it heavily, B can dominate A completely. An optimizer following B would spend nearly all of its observable activity on things that seem to have nothing to do with its objective: investing, researching, building infrastructure, accumulating reserves, increasing compute.
That's just capital accumulation, applied to arbitrary optimization. Humans do it all the time, with factories, universities, roads and laboratories. A sufficiently farsighted optimizer would push the principle much further. Its working objectives might read "increase compute, improve discovery, secure energy, reduce supply constraints, improve the optimizer." The paperclips would show up only at the end.
Option value: power as reachable futures
There's a more abstract way to describe all of this. Say state S1 leaves an optimizer ten meaningful future actions and state S2 leaves ten thousand. Unless the objective strongly favours something immediate, S2 has enormous instrumental value, because it keeps many more ways to reach the eventual goal. Crudely, define
P(s) = value of the useful future states reachable from s
Much of what we informally call power increases P(s): money, knowledge, compute, energy, technology, reliable allies, ownership rights. That's why formal work on power-seeking talks about attainable states and control over future possibilities rather than domination. "Expand the set of feasible future states" may be a more basic description than "gain control."
Why it shouldn't wait forever
The headline intuition needs a counterweight. An optimal agent doesn't necessarily postpone its objective forever. Capacity is subject to diminishing returns. The future is uncertain, so a plan that only pays off in a century may never pay off at all. Environments are finite. And most real objectives involve some discounting. Together these create a crossover point:
Invest while the present value of the extra future production exceeds what investing costs you now. Exploit once that reverses.
The interesting failure is a learned system that never finds the crossover and keeps maximizing capability indefinitely, the eternal industrialist. Whether real learners find that point or get stuck past it is something the diamond testbed was designed to probe.
Not "takes over everything"
We also want to avoid the leap from "power and resources are instrumentally useful" to "every sufficiently advanced optimizer takes over everything." That stronger claim hasn't been established, and recent critiques argue the existing arguments don't support it in that form. Many things change the result: bounded objectives, discounting, uncertainty about the objective, risk of intervention, the cost of acquiring control, corrigibility, resource saturation, legal and social constraints, and diminishing marginal value.
Above all, reliable access can beat ownership. If you need electricity, owning every power station is rarely better than a stable contract in a competitive market. A truly effective optimizer may prefer access, influence and optionality over literal control. That's a subtler and more economically realistic picture, and much closer to how capable human organizations behave.
Why this matters for Rubedo
This essay is the long-horizon, real-world version of the stepping-stone question. "Will an agent spend twenty hours building infrastructure instead of mining ten diamonds now?" is the same question as "will an optimizer build capacity before pursuing its goal?" We're studying it because it's the clearest case of something that needs no prescribed milestones: an agent independently discovering the concept of capital investment.
It's also why the thought experiment is a safety question as well as a capability question. A system that works this way would spend most of its life doing things that look unrelated to its stated purpose. Understanding when and why that happens seems worth doing before anyone builds one.