Why a static agent first
After working through the cost of a generation, we set reproduction aside and asked a simpler question. What would a single, non-reproducing learning agent need in order to take part in a world of other agents, markets and public information? Two things stood out: an internal state, so it can remember and reason over what it has seen, and an action space far beyond "move left, move right", so it can buy something, message someone, or choose a counterparty. Communication is what turns a collection of such agents into a society.
Memory and recurrent cognition
Memory can't be an afterthought for an adaptive system. We've found it useful to separate several kinds: private working memory, recurrent internal state, long-term episodic memory, shared knowledge, externally verifiable records, social memory and reputation, and lineage or cultural information. For a single agent there's a rough ladder:
| Tier | What it is | Typical failure |
|---|---|---|
| A. Recurrent policy | A GRU or LSTM carries state between steps. Cheap baseline. | The memory bottleneck makes long episodes hard to hold onto. |
| B. Recurrent core + external memory | A key–value store the agent learns to write to and read from. | Memory pollution, retrieval collapse, runaway writes. |
| C. World model + planner | Predicts consequences and plans over them. | Model bias, compounding error, expensive planning. |
Tier B looks like the practical starting point, with a small world model added later. Memory brings a population-level problem too. If agents copy one another's memories indiscriminately, the population can collapse into an intellectual monoculture (see Collectives).
Very large action spaces
Real participation means actions like "trade asset 4,182 at this price" or "send this message to that node." A single flat output layer over every possible action doesn't scale. Each output needs its own learning signal, most are rarely chosen, and the space grows combinatorially. Our first instinct was to have the output layer call a function that invokes a secondary, specialized model to complete the action. That turns out to sit inside a family of established approaches:
- Factorized or autoregressive actions. Emit an action as a sequence, such as verb, then target, then amount, each from a small head, with invalid continuations masked out. It's flexible, but credit assignment across the sequence is hard.
- Parameterized actions. Choose an action type, then output the continuous or structured parameters for that type. This is a clean interface for "functions", but the policy can collapse onto whichever type is easiest to learn.
- Retrieval and pointers. Rather than a softmax over every node or asset, build a candidate set and point into it. This scales to millions of entities, but if retrieval misses the best option, the policy can never pick it.
- Hierarchical options. A manager chooses a skill (reproduce, message, move, trade) and the skill executes over several steps until it ends. This matches our "secondary model" idea most closely. It's powerful for long horizons, but it's sample-hungry if trained purely from sparse reward.
- Mixtures of experts. Soft routing to sub-policies. It's differentiable and stable, but at risk of the router always picking one expert.
- Programmatic actions. The policy writes a well-typed call in a small action language and an interpreter executes it. That gives near-unlimited expressiveness, but any bug in the interpreter becomes the world's physics.
If a high-level choice hands off to a secondary model that also makes stochastic choices, those choices must be accounted for in the probability of the action that was actually taken. Otherwise the learning signal is optimizing the wrong thing. Composing models is fine, but the composition has to stay a single coherent policy.
Thinking versus acting
We also wanted agents that can keep updating their internal state without acting: thinking further, waiting for new information, then acting. That's something like extended reasoning, chosen by the agent itself. The catch is that free thinking is a loophole. If extra internal steps cost nothing, agents will either think forever or learn degenerate strategies. Thinking has to be capped or priced.
That leads to a broader formulation we like. Put thinking, remembering, communicating, acting, learning and reproducing on one shared, scarce resource budget. Intelligence then becomes partly a question of deciding where computation is worth spending. It's the same question as in the compute page, asked inside one agent.
Communication that costs something
Communication shouldn't be a free, magical side channel. When it has costs and consequences, agents face real strategic choices: whether information is worth sending, to whom, how much bandwidth to spend, whether the other party can be trusted, whether to keep something private, whether deceiving is profitable, and whether a reputation is worth building.
- Cheap talk (free, unenforced messages) tends to be ignored or exploited.
- Costly signalling (messages carry a cost or a stake) can make honesty sustainable. That's the logic of the handicap principle in biology and signalling in economics.
Our working design: messages are embedding vectors (one or several) that cost something to send, recipients are free to ignore them, and agents learn whom to listen to through trust and relevance filtering. The failure to watch for is emergent collusion: protocols that win reward while being useless or harmful to everyone else.
LLM agents: policy, harness and experience
For a language-model agent, memory, retrieval, tools, retries and stopping rules form a harness around the policy. ReAct demonstrates interleaving reasoning and action in selected tasks. Running that loop with frozen weights is distinct from agentic RL, which trains a policy from interaction trajectories. We should measure both the model and the surrounding system.
Investigation. Freeze a model and compare direct answers, independent sampling and tool-assisted revision with equal total budgets. Then train from checked trajectories and test under the original inference limit. Track tool failures, recovery, memory provenance and the gap between generating a correct candidate and selecting it. Long-horizon credit assignment, unreliable tools and adversarial observations remain open problems. The frontier agenda develops this comparison.
What we don't know
- How persistent memory should work, and how to stop shared memory from homogenizing a population.
- How agents should decide what information is worth communicating, and to whom.
- Which large-action-space design keeps learning stable when the set of entities (nodes, assets) keeps changing.
- How to price internal computation so agents think enough but not endlessly.
- How trust, deception, reputation and coalitions evolve once communication carries real costs.