Two ways to fail
Picture a population of agents. One discovers branch mining and starts earning steady reward. Several others are messing about with redstone and earning nothing. Under pure fitness-based selection the experimenters are eliminated, even though one of them was on a path to a strategy a thousand times better.
Now picture the opposite. Selection rewards only novelty, so the population fills with endless variations, almost none of them useful. Memory and compute vanish into redundant difference.
Selecting only for current reward kills future options. Selecting only for novelty accumulates irrelevant variation. The real challenge is retaining useful optionality with finite memory and compute, while not knowing in advance which discovery will become valuable.
What the literature offers
Literature
Novelty search
Lehman and Stanley's novelty search drops the objective as a selection criterion and rewards behaviour that differs from what's been seen before. In deceptive problems this can reach the goal more reliably than chasing the goal directly. It's a striking result, but it rests on a choice: you need a way to measure how different two behaviours are.
Quality-diversity
Quality-diversity methods such as MAP-Elites keep a repertoire of the best solution found in each behavioural niche, rather than one global best. The archive is the preservation mechanism. Diverse alternatives survive because they're the best of their kind, not because they're the best overall.
Archives and return
Go-Explore keeps an archive of interesting states and explicitly returns to them before exploring further, so progress isn't lost to random wandering. POET goes further, pairing generated environments with agents and transferring solutions between them, so a solution that's useless in one place can become a stepping stone somewhere else.
The hidden assumption: what counts as "different"?
Every one of these methods needs a behavioural descriptor or a distance. That's where the designer's theory of progress quietly creeps back in. If you describe Minecraft behaviour by "distance travelled", "blocks broken" and "redstone used", you've already decided that redstone is a meaningful axis of variation. That's a milestone in disguise.
Learned descriptors may reduce this dependence. Some recent work proposes deriving them automatically from what policies actually do, for example from occupancy measures. We haven't verified those claims closely, and the experiments we know of are in much simpler domains. And any learned representation still carries the assumptions of whatever trained it.
Biology has no designer choosing behavioural descriptors, yet it preserves an enormous diversity of lineages. Niches, geography, speciation and the plain fact that different organisms eat different things do the work. Could an environment rich enough in distinct resources and roles make the descriptor problem disappear, by letting the world itself define the niches?
Preservation inside a single learner
The same problem shows up within one agent over time, as catastrophic forgetting. Learning a new skill can damage the representations an old skill relied on. Systems that store skills symbolically, like Voyager's library of executable code, sidestep this. A purely neural learner gets no such guarantee. Something analogous would have to emerge, or be built in as a general mechanism rather than a domain-specific one.
There's a related and less discussed failure, loss of plasticity: networks trained continually can gradually lose the ability to learn at all. Preserving competence and preserving the capacity to keep changing are both necessary, and they pull against each other.
Archives for frontier reasoning and discovery
Hypothesis An archive of checked programs, proof strategies or tool workflows could retain stepping stones that immediate reward selection loses. Quality-diversity search offers a mechanism, but its descriptors determine what counts as different. Distinct wording is weak evidence of distinct strategy.
Investigation. Compare bounded quality-diversity, novelty, reward-ranked and random archives around the same frozen model and evaluator. Measure downstream transfer, reuse, retention and total cost, alongside archive coverage. Remove the archive or replay its entries to test whether it caused later gains. Track provenance, deduplicate behavior and recheck candidates when their dependencies change. The archive and curriculum pilot connects preservation to frontier search.
What we don't know
- What finite retention criteria could work without a human-designed definition of relevance.
- Whether a rich environment can supply its own niches, removing the need for hand-built descriptors.
- How a population should allocate scarce resources between exploiting what works and sponsoring what might.
- How to retain competence over long lifetimes without freezing the learner in place.