Surprise is not enough
The simplest computational curiosity rewards an agent for observations it failed to predict. That works surprisingly often. It also has a famous failure, the noisy TV problem: put a screen of random static in the environment and a surprise-seeking agent will watch it forever, because static is unpredictable by definition. High prediction error can reflect irreducible noise as easily as learnable structure.
The fix usually proposed is to reward learning progress, meaning the improvement in prediction or competence, rather than raw error. Static never gets more predictable, so it stops being interesting. Learning progress is a candidate mechanism for a self-generated curriculum: practice where you're improving, move on when you plateau or when something is hopeless.
But it isn't solved. Measuring genuine progress, as opposed to memorizing the latest experience, takes care. Hard prerequisites can stall progress for a long time before the payoff, and an agent that leaves whenever progress stalls may abandon exactly the investigations that matter. This links straight back to stepping stones.
A vocabulary problem
Literature
Oudeyer and Kaplan's typology of intrinsic motivation is useful mostly because it shows how many different things get called "curiosity": novelty, surprise, competence, learning progress, information gain. These come apart in practice, and the psychological sense of "intrinsic" isn't the same as "internally computed." Gottlieb and Oudeyer draw another useful line, between sampling information for a known purpose and exploring to discover what is worth learning. Rubedo cares mostly about the second.
Autotelic agents, in Colas, Karch, Sigaud and Oudeyer's survey, go further. They represent, generate, prioritize and pursue their own goals in order to build repertoires of skills. That's close to the "self-generated developmental goal" we're interested in. Still, learned goals depend on supplied goal representations and learning machinery, so goal generation isn't the same as unconstrained autonomy.
Hypothesis: the drives themselves could adapt
Hypothesis
Most systems hand-design their intrinsic rewards: a novelty term, a competence term, a weighting between them. Ours is a different hypothesis. The mechanisms that decide what to attend to, practice, remember or explore could themselves adapt through learning or evolutionary selection.
Biology suggests this is possible. Nobody designed the specific things animals find interesting. Selection shaped those tendencies because, across many lifetimes, the organisms that explored in certain ways did better. Neuroscience increasingly suggests that dopamine and related systems are richer than a single scalar reward signal, which is a reason to be cautious about simple "the brain does RL" analogies.
This doesn't remove the need for assumptions. Something still decides which drives survive, the outer selection criterion. But it moves the designer's choice up a level, from "what to explore" to "what makes an exploration strategy good over a lifetime." That may be a much more general choice.
A drive that produces excellent behaviour quickly may be worse at finding transformative alternatives over a long lifetime. If we select exploration strategies on short evaluations, we'll select against exactly the patient curiosity we want. Making drives evolvable doesn't solve that credit-assignment problem; it moves it.
Exploration without a task
One promising thread: learn about the world first, with no downstream objective, and adapt later. Plan2Explore seeks out experiences where an ensemble of learned dynamics models disagrees, and the resulting world model can then serve tasks it never saw. Contrastive Intrinsic Control learns a repertoire of diverse behaviours without any reward. Neither connects discovery to the eventual objective on its own, and both are demonstrated in domains far simpler than an open world. But they hint at a learner that finds what in its world is worth understanding without being told.
What we don't know
- Whether learning progress can be estimated reliably enough to steer exploration across long, stalled prerequisites.
- How intrinsic drives should trade off against an external objective, and who decides the exchange rate.
- Whether evolving drives (rather than designing them) yields more general exploration, at a comparable total compute budget.
- What the right outer criterion is for selecting a drive, when its benefits may take a lifetime to show.