Open questions / 09

When does a population become a higher-order optimizer?

The basic unit of superintelligence may not be an agent. It may be an ecology of agents that learns to organize itself into increasingly powerful higher-order optimizers.

Open questionHypothesis

Where Rubedo started

Rubedo's oldest question was about populations. If many adaptive agents share an environment with scarce resources, communication, competition, cooperation, memory, trust and the possibility of deception, what larger structures emerge? We wanted to see specialization, coalitions, conventions, persistent roles, and strategies that only make sense given who else is around.

The underlying hope was that local learning + evolutionary selection + economic pressure + social organization might add up to population-level intelligence. The unit of analysis would be the ecology rather than the agent. Human civilization is the existence proof: enormous capability emerging from many bounded agents connected by communication, institutions, markets, culture and accumulated knowledge.

A practical simplification later dropped reproduction in favour of a static multi-agent population: agents with recurrent memory, structured actions like trade and messaging, costly communication, and trust filtering (see Agents). It was meant as a feasibility step, a small toy market of perhaps 5 to 20 agents with hidden environmental regimes, public and private information, assets and incomplete information. Agents in such a market might discover information sharing, specialization, reputation, deception, arbitrage, coalitions and collaborative inference. It hasn't been built yet (see Experiments), and the question behind it has only grown.

Cooperation without altruism

Working synthesis

A distributed optimizer doesn't need its parts to share a goal. Suppose agents A1, …, An each pursue their own utility Ui. What matters is whether there are instrumental resources that nearly all of them benefit from: compute, energy, knowledge, productive capacity. An AI lab wants cheaper compute, a manufacturer wants cheaper energy, a fund wants better forecasts. They disagree about almost everything else, but increasing certain kinds of capability helps nearly all of them.

Now give agents different strengths. One is good at discovery, one controls capital, one manufactures, one moves goods, one has market information. None has to give up its objective. Each can find that coordinating beats going alone. Cooperation becomes an instrumental strategy, and specialization follows from comparative advantage:

Independent agents→ Exchange→ Specialization→ Persistent relationships→ Coalitions→ Institutions

Eventually the institution matters more than any member. A company doesn't stop existing when an employee leaves. It keeps its goals, capital, knowledge, contracts and procedures through continuous turnover. We already have empirical examples of higher-order agents emerging from lower-order ones. Markets go further: there may be no central controller at all.

Levels of emergence

Individual optimizer ↓ Coalition: a few agents with shared interests ↓ Institution: persists through turnover of its members ↓ Economic ecosystem: many institutions, coordinated by prices and contracts ↓ Civilizational optimizer?

Something strange happens on the way up. Lower-level parts can want things that conflict with what the larger system appears to want. A worker prefers leisure, a firm wants output, a consumer wants low prices, an investor wants returns. Yet their interaction can still steadily increase total productive capability, an objective nobody chose. That phenomenon is exactly what we'd want Rubedo to help explain. The larger version of this argument is in Civilization as an optimizer.

Culture: escaping rediscovery

Literature

Cumulative culture is how knowledge outlives the individual who found it: preserved, transmitted, modified and recombined across people and generations. Tomasello calls it the ratchet effect, and Henrich argues it's the real secret of human success. No single person could rediscover what a culture knows. Recent artificial-life work treats culture as a new dimension of open-ended evolution, and as a way open-endedness itself can evolve.

For an artificial population, transmitted knowledge could prevent each agent from rediscovering everything. But transmission brings its own problems: fidelity, abstraction, dependence on teachers, and incentives to share at all. A population that shares mistaken predictions can amplify error just as efficiently as knowledge.

The monoculture problem

There's a sharper version of this. If agents copy one another's memories and conclusions freely, a population can collapse into an intellectual monoculture: everyone believes the same things for the same reasons, and independent exploration stops. A healthy population needs mechanisms that preserve independent exploration while still letting discoveries spread. That tension runs through all of Rubedo. Things we'd want to measure in any multi-agent experiment:

A related question we find hopeful: can a population of specialists collectively become more general than any one member?

Major transitions

Maynard Smith and Szathmáry described a small number of major transitions in evolution: from replicating molecules to cells, single cells to multicellular life, individuals to societies, and the arrival of language. In each one, entities that could once replicate independently became parts of a larger whole, and the way information was transmitted changed. Adaptive change sometimes alters the unit of organization rather than improving the existing one.

Applying that to artificial agents is an analogy, and one that needs its own definitions and evidence. Agents collaborating isn't automatically a major transition. But "under what conditions does a collection of agents begin behaving like a higher-level agent?" may be the most important question on this site.

Environments that agents change

Agents don't just adapt to environments, they reshape them. Niche construction (beaver dams, cities, markets) changes future opportunities and selective pressures. In open-ended systems, the environment and its inhabitants co-evolve. Multi-agent research calls this an autocurriculum: each strategy creates a new problem for everyone else, and the curriculum generates itself. OpenAI's hide-and-seek experiments are a vivid example. Agents discovered tool use in successive waves, each one a response to the other team's last innovation.

A caution: evolution isn't directed toward intelligence or complexity. Simple strategies often stay successful. Artificial complexity has to be judged against capability and cost, not assumed to be good.

Synthetic worlds as wind tunnels

Should Rubedo build artificial worlds at all, when the real economy already contains richer pressures than anything we could design? We think yes, for a specific reason. Studying real society alone has an n = 1 problem. You can't rerun history with different property rights, remove money, or change when computers were invented. A controlled world can, and it lets us ask precise questions:

The important change is where the mechanics come from. Instead of inventing arbitrary game rules and hoping they say something about reality, derive the mechanics from real economic and evolutionary phenomena. Real civilization generates hypotheses, and controlled worlds test mechanisms.

Testing coordination and shared memory with LLM agents

Hypothesis Shared memory might turn individual attempts into cumulative knowledge, but it can also spread a mistake and erase useful diversity. Agent-scaling experiments report that coordination helps or hurts depending on the task and architecture; they do not support a universal advantage for more agents.

Investigation. Compare a single agent with independent workers, a coordinator and a shared-memory group under the same total computation and storage budget. Separate the effects of roles, communication and memory. Require provenance and independent checks for accepted artifacts, keep unverified proposals distinct, and revalidate stale records. Test both separable and sequential tasks, including poisoned-memory recovery.

The measure is externally checked task success and reuse by fresh agents, alongside duplicate work, communication cost and correlated failures. Agreement is not evidence of independence when agents share a base model or copy the same record. See the coordination pilot.

What we don't know