Open questions / 01

How does a learner value what doesn't pay yet?

The path to a much better strategy often runs through tools, knowledge or infrastructure that produce nothing at first. A system that only follows immediate reward will never take that path.

Open questionWorking synthesis

The problem

Imagine an agent whose only goal is to collect diamonds. Early on it learns that mining at the right depth with an iron pickaxe gets results: a steady, reliable trickle. Somewhere far beyond that strategy is another one: build farms, automate materials, learn redstone, build machines, and eventually mine diamonds at industrial scale, thousands of times faster.

The catch is that every step toward the second strategy makes things worse at first. Time spent on infrastructure is time not spent mining. From where the agent stands, the direction of immediate improvement points away from the much larger payoff. Lehman and Stanley call these deceptive objectives: the stepping stones to a goal often don't look anything like the goal.

We think this is more than a quirk of games. It's the structure behind capital investment, scientific research, education and evolution's long detours. A productive local strategy can suppress exploration of every alternative that's initially unproductive.

Three very different kinds of milestone

A lot of confusion comes from treating "intermediate guidance" as one thing. We think there are at least three, and the difference is the whole question.

  1. Human milestones. The designer rewards a chosen intermediate event: r(iron) = +1. This works, but it smuggles the answer in. The system is told what the path is and only has to learn to walk it. For the discovery problem we care about, we set this aside.
  2. Learned instrumental value. Experience makes an intermediate state valuable because it reliably comes before success. If iron-holding states keep leading to diamonds, ordinary temporal-difference learning raises their value. No one named iron; the agent's own experience did. But this only works if the full path has been experienced at least once.
  3. Self-generated developmental goals. The agent decides for itself to practice reaching some kind of state, or to investigate some phenomenon, before it knows whether that helps. This is the most interesting kind and the least demonstrated.

The second and third kinds needn't correspond to anything a human would name. If an agent's internal goals turned out to look like "iron" or "furnace", that would be a discovery, not an assumption.

Why this matters for existing results

DreamerV3's well-known Minecraft result learned to obtain diamonds from pixels, which is impressive. But the task rewarded twelve intermediate milestones (log, plank, stick, crafting table, wooden pickaxe and so on up to diamond). That's the first kind of milestone. It shows that a general algorithm can learn a long prerequisite chain. It doesn't show that the chain can be discovered from a bare terminal reward. We discuss this further on the testbed page.

The zero-reward basin

Learned instrumental value has a cold-start problem. Early in learning, essentially no trajectory reaches the goal by chance. If every run returns zero, there's no gradient telling the learner which behaviour is "closer", and nothing for value to propagate backward from.

This is the classic sparse-reward exploration problem. Go-Explore attacks it by archiving interesting states and deliberately returning to them. Plan2Explore attacks it by seeking out experiences its world model disagrees about. Both help, and neither solves it at the scale of a real technology tree. Each also needs some notion of what counts as "interesting", and if humans define that, the milestone problem comes back in disguise.

Why long horizons are hard even with a world model

A common response is to give the learner a world model and let it plan. That helps a great deal, but the arithmetic is unforgiving. With a discount factor of γ = 0.997, a reward 10,000 steps away contributes about 0.99710000 ≈ 10−13 of its value. Raising γ increases variance instead. And rolling a learned model forward for very long horizons compounds small errors until the agent finds plans that look great only because the model is wrong. Practical systems imagine short distances and let a learned value function bootstrap the rest.

The upshot: a state like "base full of unused iron and redstone, zero diamonds" can be undervalued simply because the agent has never cashed in from that state. The world model may understand it perfectly well.

Capacity-building is a known phenomenon

Literature

The intuition that very different goals all reward building capacity first has a name. Bostrom called it instrumental convergence: resources, cognitive enhancement, technology and self-preservation help with almost any objective. Turner and colleagues gave a formal version for Markov decision processes: under certain symmetry conditions, optimal policies tend to prefer states that keep more future options reachable. Empowerment, roughly the mutual information between an agent's future actions and its future states, is one candidate measure of that "future controllability."

We read these as a formal backbone for the stepping-stone intuition, with caveats. Turner's results hold in the formal setting studied, and extending them to large learned agents is an analogy, not a theorem. The broader argument is explored in The patient optimizer.

When should a learner stop investing?

There's a counterweight we don't want to lose. An optimal agent doesn't necessarily postpone its objective forever. With diminishing returns, a finite or uncertain horizon, or discounting, there's a crossover point where another unit of investment is worth less than cashing in. Invest while the present value of the extra production exceeds what investing costs you now; exploit once it doesn't.

Whether a learned system would find that crossover, rather than getting stuck maximizing capability forever (the "eternal industrialist"), is an open empirical question. Over-weighting empowerment produces exactly that failure.

What we don't know