The paperclip problem assumes the hard part away
The classic thought experiment hands the optimizer its terminal objective (maximize paperclips) and asks what follows. Our discussions keep returning to a different, more fundamental question:
What happens when an increasingly capable civilization becomes an optimization process before it has agreed on what it is ultimately optimizing?
An optimizer can't exist without some criterion of preference. If nothing distinguishes one trajectory from another, there's nothing to optimize. But the criterion needn't look like a final goal. It could be a mix of knowledge, control, survival, productive capacity, energy and preserved future possibilities. And it may not be written down anywhere.
It's already happening
The usual story is: society builds AI, and AI becomes an optimizer. The real sequence may look more like this:
Nested optimization processes are already everywhere. Evolution selects organisms. People pursue mixtures of preferences. Firms pursue profit, growth and survival. Markets allocate capital toward expected returns. Governments pursue political, military and economic goals. Science searches hypothesis space. Algorithms already optimize logistics, advertising, trading, manufacturing and design.
AI isn't introducing optimization into a world that lacked it. It's being inserted inside each of these loops. People are using it right now to optimize capital returns and production. So a transition to a "super-optimizer" might not happen when someone launches a single system. It might happen as thousands of optimization processes slowly become interconnected:
At some point it becomes reasonable to ask whether civilization itself should be modelled as the agent, as a mathematical description and not only as a metaphor.
Instrumental convergence at the scale of civilization
Hypothesis
If civilization is an optimizer, it probably doesn't have a clean terminal objective. Selection pressure may still favour a set of instrumental ones. Systems that acquire resources tend to persist. Systems that model their environment well outperform those that don't. Systems that keep their options open adapt better. Systems that improve themselves outpace those that stay static.
So a meta-objective could emerge, roughly more capability, knowledge, resources, optionality and future influence, without anyone ever choosing it. Actors who disagree deeply about what matters may still agree, in their behaviour, on more energy, more compute, better prediction, better technology and more productivity.
Civilization could converge on a direction of optimization before converging on a philosophy of what optimization is for. That arguably describes the present.
An inversion
The textbook picture puts values first:
A technologically accelerating civilization may experience something closer to:
In other words, the optimizer may partly determine its own goal through selection. Whatever organizations, algorithms, cultures, economic arrangements or artificial agents reproduce successfully become more represented in the future. That looks much more like evolution than like classical optimization.
Writers from very different traditions have circled this. Bostrom's essay on the future of human evolution asks whether competitive selection could erode things we value. Scott Alexander's "Meditations on Moloch" describes competitive dynamics producing outcomes no participant wants. More recently, Critch and Russell's taxonomy of societal-scale AI risk and Kulveit and colleagues' work on gradual disempowerment both argue that the most important failures might be diffuse and systemic, not one dramatic event.
No single machine required
Popular imagination pictures a superoptimizer as one AI controlling everything. That may be the least realistic version. A more plausible path is a distributed optimizer made of partially aligned agents: billions of people, millions of organizations, markets, AI systems, automation and infrastructure, all coupled by feedback. No component understands the whole. Yet collectively the system gathers energy, mines resources, manufactures computers, launches satellites, does research, reallocates capital and improves its own computational infrastructure.
There's already a primitive self-improvement loop here. The economy produces computers, computers improve economic optimization, a better economy produces better computers, and better computers improve AI. Nobody has to program "acquire more power so you can acquire more power." Competition can produce something like that behaviour on its own, because organizations that don't invest in productive capacity tend to lose to those that do.
Eric Drexler's "Comprehensive AI Services" made a related argument from a different direction. General capability might arrive as an ecosystem of specialized services rather than as one unified agent. Hayek's point that prices coordinate knowledge no one holds in full suggests how such an ecosystem could coordinate without a centre.
What steers it?
This is where the picture turns from fascinating to worrying. If acquiring resources and gaining optimization capacity reinforce each other, a sufficiently unconstrained optimizer, or network of optimizers, could come to treat influence over capital, production, energy and institutions as instrumentally useful. That's the classic instrumental-convergence argument, applied to a system nobody controls.
If resource acquisition and optimization capacity reinforce one another, what constrains the direction in which the resulting system optimizes?
That's why the objective can't simply be "become more capable." A civilization-scale optimizer needs mechanisms that keep growing capability tied to things people actually value. Our tentative hope (that useful output attracting resources might do some of that work) is discussed in Incentives. We haven't solved this. Working on Rubedo has mainly made clearer how much it matters.
Three layers, one question
This gives us a way to organize the whole of Rubedo:
- Optimization theory. What optimizing means when the landscape is enormous, changing and partly unknowable.
- Optimizer ecology. What happens when many optimizers interact, cooperate, compete and select one another.
- Civilizational meta-optimization. What happens when those optimizers are coupled tightly enough that the whole starts to resemble a higher-order optimizer.
At the top of that stack is a question we can't answer and find hard to stop thinking about:
What is the objective of the optimizer composed of all optimizers?
It may not have a predetermined answer. It may be exactly what the system is in the middle of discovering.
What Rubedo can contribute
This reframing doesn't make the multi-agent research obsolete. It explains what it's for. Instead of asking "what reward function should we give an intelligent agent?", the deeper question becomes what kinds of objectives emerge when optimizers are themselves subject to competition, mutation, learning and selection?
That suggests an experiment we find compelling. Don't specify the grand objective at all. Give agents local incentives, scarce resources, learning, communication, perhaps reproduction, and the ability to modify parts of their environment. Then watch for larger-scale optimization structures. The interesting result wouldn't be what they optimize. It would be whether a higher-order optimizer appears at all, and under what conditions. See Collectives for the concrete questions.
The question touches alignment, economics, evolutionary theory, control theory, game theory, philosophy of mind, political economy, artificial life and open-ended learning. We don't expect to answer it. We'd like to make parts of it testable.