Why this is central to decentralized AI
Rubedo's interest in decentralization has always been partly about participation. We imagined independently hosted agents and interacting populations, with participants rewarded for being genuinely useful. That immediately raises the question every open network faces: who decides what "useful" means, and what stops participants optimizing the judge rather than the job?
Why we stopped decentralizing one mind
Earlier direction
Rubedo's earliest technical direction was to decentralize something close to a conventional neural network: running parts of a model across independently operated machines, and eventually using cryptographic and zero-knowledge proofs to verify the work of untrusted nodes. The problems kept piling up. There was communication latency, synchronization, stragglers, adversarial machines, proof-generation overhead, verification complexity, and the tight coupling that gradient-based training needs. More importantly, it was turning Rubedo into a distributed-computing and cryptography project rather than an intelligence research project.
So we pivoted toward locally sovereign agents. Each machine runs a complete learner, and agents interact only when it's useful to them.
Don't decentralize one mind. Investigate whether intelligence can emerge from a decentralized ecology of minds.
That moved the hard problem. Proving that computation happened isn't the same as showing that it was worth doing, and the second is the problem this page is about.
Where does real value come from?
One of Rubedo's oldest unresolved problems: a closed economy in which agents only pass tokens back and forth tends to be dominated by predation. Stealing is easier than creating. An ecosystem becomes interesting only when agents can create value. Things that naturally create positive-sum opportunities include uncertainty, information asymmetry, specialization, complementarity, coordination, production, discovery, and above all external demand. The original hope was that useful intelligence would be valuable to people outside the system, so that a productive ecosystem could sustain itself rather than just circulating tokens internally.
Hypothesis That led to an idea we've called Schelling-point alignment. People already coordinate around things such as gold, money, companies and markets, which have value because other people value them. Perhaps an artificial ecosystem could be partly aligned by the same force. Systems that produce outputs people actually find useful acquire resources, so useful lineages and organizations grow while useless ones starve. It's an appealing idea and entirely unproven. It doesn't remove Goodhart's law, manipulation, monopoly, fraud, coercion or other pathological equilibria.
Goodhart, in several flavours
Literature
"When a measure becomes a target, it ceases to be a good measure." Manheim and Garrabrant break Goodhart's law into distinct mechanisms, and the distinctions help:
- Regressional: the proxy and the goal are correlated, but pushing hard on the proxy selects for noise.
- Extremal: the relationship that held in normal conditions breaks down at extremes.
- Causal: intervening on the proxy doesn't cause the goal, it just correlated with it.
- Adversarial: other agents deliberately exploit the gap between proxy and goal.
In a decentralized system with many optimizers, all four show up, and the adversarial kind is constant. Reinforcement learning has produced many examples of specification gaming, agents satisfying the literal reward in ways their designers never intended.
Evaluators that degenerate
A specific worry from Rubedo's earlier work: evaluators that adapt alongside the agents they judge can drift into self-reinforcing equilibria, rewarding what's familiar, what flatters them, or what's easy to check, rather than what's good. Co-evolving judges and players can lock each other into a stable but useless pattern.
The same issue appears in the curiosity question. An agent's own intrinsic reward is an evaluator too, and the noisy-TV problem is a kind of internal Goodhart failure.
Our stance: a design problem, not a stop sign
It's easy to look at Goodhart's law and conclude that open, decentralized optimization is hopeless. We don't. Evaluator manipulation is a design problem. Directions we find worth thinking about:
- Grounding in consequences. Reward outcomes that are hard to fake because they're physical, delayed or independently checked.
- Markets as evaluators. Prices aggregate many independent judgments, which is Hayek's point about knowledge in society. They can be manipulated too, but they're harder to capture than a single judge.
- Plural, rotating and adversarial evaluation. Make gaming any one judge less valuable.
- Cost. If communication, claims and participation carry real costs, cheap manipulation becomes expensive.
- Reputation and trust. Earlier designs included trust and relevance filtering on messages, so agents learn whose information to rely on.
- Stake-backed claims and real demand. Agents put resources behind their claims and lose them if they're wrong. The ultimate judge is whether actual people want what's produced.
The framing we've settled on: make the measurable target increasingly hard to tell apart from the thing we actually care about. None of these is a solution, and each fails in known ways. Turning "real usefulness" into a machine-evaluable signal is still an open problem.
Incentives at civilizational scale
The same problem appears at the largest scale we think about. If civilization is becoming a distributed optimizer, its objectives are shaped by what gets rewarded: profit, growth, survival, influence. Those are proxies for what people actually value, and the gap between proxy and value is Goodhart's law applied to a whole society. Recent work on gradual disempowerment asks whether incremental automation could erode human influence over these systems with no single dramatic failure. We explore this in Civilization as an optimizer.
Verifiable rewards and the objective behind them
A verifier can make feedback cheap without making the objective complete. Tests can omit important behavior; a formally valid proof can establish the wrong specification; a learned judge can reward convincing errors. Repeated search or RL against one checker increases pressure on its blind spots. That is a concrete Goodhart risk for the frontier learning agenda.
Investigation. Freeze evaluation rules and splits, separate search feedback from a final audit, limit repeated validation access and inspect false acceptances. Use counterexamples, mutation tests and independent checks where feasible. Keep accepted artifacts distinct from unverified claims and record the human contribution to problem selection, formalization and repair. Correctness alone establishes neither novelty nor significance.
For adaptive controllers and agent populations, keep evaluator code and final tasks outside candidate write access. Restrict tools and execution, preserve provenance and reversible checkpoints, and test collusion, memory poisoning and correlated judge errors. These controls can expose failures; they do not certify alignment or guarantee that a stronger optimizer remains safe.
What we don't know
- How to turn participation and usefulness into signals a machine can evaluate without the signal being captured.
- Whether co-evolving evaluators can be kept honest, and what keeps them from collapsing.
- What reward designs support both real usefulness and viable participation for resource-limited agents.
- What prevents predatory equilibria, and what generates genuinely positive-sum external value.