acceptodds
Under review as a conference paper at ICLR 2027

Geometric Regularization of Learned Behavior Spaces in Unsupervised Quality-Diversity Optimization

Abstract

Discovering increasingly capable behaviors can require preserving solutions whose value only becomes apparent when later discoveries build on them. A central challenge is thus retaining these stepping stones without specifying in advance which behavioral differences will matter for future progress. Quality-diversity algorithms pursue this goal by maintaining a repertoire of distinct policies that the search can revisit and improve. For example, learning a latent representation of behavioral attributes directly from actions of a robot removes the need to specify the behavioral dimensions of any given policy that distinguish meaningfully different solutions from one another. Prior works, like AURORA, train autoencoders on action trajectories to find these important dimensions of variation. But autoencoder reconstruction loss alone does not uniquely determine the latent-space distances between potential solutions. Here, we introduce AURORA-VC, which adds variance and covariance penalties to online trajectory reconstruction. These penalties encourage variation across descriptor dimensions and discourage correlations between behavioral descriptor axes independently of the selected loss or fitness functions used later to assess solution quality. With both PGA-AURORA and AURORA, maximum fitness improves in four of ten tasks at an uncorrected paired , with task-dependent effects on repertoire quality. VC is associated with longer survival of high-fitness admissions and reuse of older parents. Survivor-selection interventions support selection as one contributor to the fitness gains. Together, these results show that descriptor learning shapes the search itself, determining which policies compete, survive, and remain available as stepping stones toward higher fitness.

Primary area:
applications to robotics, autonomy, planning
Keywords:
quality-diversity optimization, unsupervised representation learning, learned behavior descriptors, geometric regularization, neuroevolution

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6zSiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies Quality Diversity (QD) methods mainly in continuous control environments, and in the setting of neuroevolution. It follows a line of work aiming at learning a diverse collection of neural policies (called an archive), the diversity criterion being learned automatically instead of being human-specified like in ordinary QD methods. Here the diversity criterion comes from the distance between latent vectors of an auto-encoder trained to reconstruct the trajectories of the policies, the main instance of this family of algorithm is AURORA that combines MAP-elites search with the reconstruction-based behavioral descriptors. The main contribution of the work is to identify a failure mode of AURORA (low reconstruction loss doesn't guarantee diversity) and adding two regularization terms, variance-covariance (VC), inspired from self-supervised learning in order to improve the geometry of the latent space. The paper then reports improved performances in terms of max fitness and qd-score (sum of fitness of the candidates in the archive) on a benchmark composed of 10 games, the reported improvement is not uniform on all tasks, and is not a huge performance improvement. Finally the paper also tries to uncover the mechanisms explaining the advantage of their regularization.

Critical strengths

None

Critical weaknesses

Presentation

I think this should be a fairly simple paper to read, and I find the phrasing and presentation obscure, and difficult to parse, even though I'm pretty familiar with the field and related works.

  • Notations: they are not well-defined, take Equation 1 alone, ϕ\phi ψ\psi oio_i are not defined and the link with the τi\tau_i mentioned in the paragraph above is not explicit. QD score is never defined, which would make it hard for the broader ICLR community to understand the paper.

  • Maths: I don't find Proposition 1 self-explanatory, I guess it's a counter-example of the "good reconstruction makes latent codes separable" statement, but I think it deserves a more pedagogical treatment. I don't even think that statement is assumed in the community, so I don't really see the point of trying to disprove it.

  • Phrasing and narrative flow: I think the writing needs more polishing. I find the phrasing overly-complex regarding the matter of study. And I think the (acknowledged) use of LLM in writing did not help.

  • Presentation of the results: I find the presentation of the results hard to follow. In the main text, I think the relative improvement of using the VC regularization compared to vanilla AURORA/PGA-AURORA is not easy to assess in Table 1. I also think ablation studies w/o C, w/o V should be in the main text and clearly highlighted. I couldn't follow well the mechanistic analysis of why/how VC helps, Figures 6 and 7 are very information-dense with a lot of metrics defined along the way. Moreover, I think 80 pages of Appendix is a huge amount of material for reviewers. Appendix B contains a lot of theoretical results, that I haven't checked, but it is not clear to me whether or not they belong in this paper, as they are only briefly mentioned in the main text.

Significance:

  • Significance of the method: I think this work is mainly incremental, take AURORA/PGA-AURORA, add a regularization and see if/why it helps. It is not rejecting grounds in itself, but it calls for a paper with a better presentation and clearer advantage compared to previous works.

  • Significance of the evaluation: I don't think the benchmark used is very significant, I know it's been used in prior works, but it kind of fails to demonstrate performance on the tasks QD actually cares about: using the archive for robustness, hierarchical RL or any interest task. I think the broader community of ICLR would be more interested in these downstream metrics than QD-score/fitness alone, that are only proxy measures in the end.

Minor

  • In the abstract you say:

Prior works, like AURORA, train autoencoders on action trajectories to find these important dimensions of variation.

If I'm not mistaken, you probably meant state trajectories.

  • Overlapping Text/arrows in Figure 1 and 6.
Questions for authors

I don't think the rebuttal could significantly change my assessment. I will still ask clarifying questions:

  • Can you clarify the demonstration made in Section 3, through Proposition 1 and Figure 1?

  • Can you clarify the importance of the mathematical analysis in Appendix B? Is it a standalone contribution or a motivation for the proposed regularization?

  • I think providing relative improvement (in %) w.r.t Aurora could make the advantage of the VC regularization clearer.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 6zsi [1/2]

We thank Reviewer 6zsi for their detailed comments and suggestions. Their requests for clearer mathematical motivation, relative improvements, and component ablations helped us make the main contribution and its empirical benefits easier to assess. Their emphasis on downstream applications also led us to expand the evaluation with frozen-archive transfer and continual-learning studies. We address all points below and have revised the manuscript accordingly. Additions to the manuscript are marked in blue. Section numbers cited below refer to the revised manuscript.

We acknowledge that the motivation, mathematical analysis, and experimental findings needed a clearer presentation. We have rewritten the main narrative, made effect sizes and component comparisons explicit, and added downstream evaluations of the archives.

Downstream Usefulness of the Archives

We thank the reviewer for emphasizing the practical uses of QD archives, including robustness and hierarchical control. We have expanded our evaluation to examine robustness under changed operating conditions, connecting the benchmark results to the usefulness of the discovered repertoires. A repertoire can provide alternative policies when the original conditions change, as demonstrated for damage recovery by Cully et al., Robots that can adapt like animals (Nature, 2015). This motivates evaluating both the competence already available in the archive and its value for further learning.

We added frozen-archive transfer and continual-learning studies in Ant-Uni, HalfCheetah, and Walker, selected for their clear primary fitness gains (Section 6.4; Appendix Q). Each study uses ten training seeds per method and environment.

Frozen probes evaluate existing policies under changed physics without further learning, using separate episodes for policy selection and fitness evaluation. At three times the original gravity, HalfCheetah fitness is approximately 3,430 with VC versus 2,647 with MSE; at twice the original actuator strength, Walker reaches approximately 4,288 versus 1,679. The complete sweeps appear in Appendix Q.5 (Figure 40).

To evaluate how the repertoires support further search, we continued learning through sudden changes in gravity, friction, torso mass, or leg strength. Under sustained changes, normalized AUC of median maximum fitness, summarized across four perturbations, is 7,120 versus 5,824 in Ant-Uni, 6,185 versus 5,766 in HalfCheetah, and 4,868 versus 2,674 in Walker (VC versus MSE; Appendix Q.4).

Together, these results connect the benchmark gains to effective policies under altered physics and higher post-change fitness AUC during continued learning. These downstream tests provide complementary evidence of the practical value of the learned repertoires alongside QD score and coverage.

Reconstruction and Archive Selection

We have clarified the purpose of Proposition 1 in Section 3: reconstruction alone leaves freedom in the distances used for archive selection. For example, doubling one latent coordinate and undoing that scaling before decoding preserves reconstruction, but can change nearest neighbors and which policies survive. The result identifies a freedom left by the reconstruction objective; it does not assume that reconstruction guarantees behavioral separability.

Figure 1 demonstrates the search consequence using a fixed pool of 8,192 PointMaze trajectories. The illustrated transformation changes 241 of the 1,024 selected policies while preserving the policies, trajectories, fitnesses, and reconstruction up to numerical error. Thus, equally accurate reconstructions can induce different archive decisions.

Appendix B provides the derivations and experimental details supporting the PointMaze study, making explicit how descriptor geometry can change archive membership while reconstruction remains fixed. It also develops supporting analysis of VC regularization, archive competition, and parent reuse to clarify the scope of our mechanistic interpretation. Section 3 now explains the role of this supporting analysis.

AuthorsOpenReview ↗

Response to Reviewer 6zsi [2/2]

Magnitude of the Gains and Component Ablations

Section 6.1 now reports relative improvements directly. In Walker, median maximum fitness increases from approximately 3,850 to 6,510 with PGA-AURORA (+69%) and from 1,660 to 2,230 with AURORA (+34%). The corresponding Ant-Uni gains are approximately 19% and 24%. We also make the task dependence explicit: Ant-Omni gains are small near its fitness ceiling, and HalfCheetah's approximately 4% maximum-fitness gain with PGA accompanies an approximately 11% lower passive QD score.

We brought the component comparison into Section 6.2. Walker fitness medians are approximately 3,850 with MSE, 2,830 with V only, 3,740 with C only, and 6,510 with V+C. Neither component alone reproduces the combined gain there or in Ant-Uni; the full task-level comparison appears in Table 9 (Appendix L).

Contribution and Implications for QD

We explicitly attribute the variance and covariance penalties to VICReg. Our contribution is to investigate how conditioning an online learned behavior space changes archive selection and the policies available for future search. We test this with both genetic and policy-gradient proposals and with a second archive-selection rule beyond DNS.

The revised Section 7 connects changes in competition to policy retention, reuse, and controlled interventions on survivors. Changing survivor identities can improve later fitness while reducing coverage, showing why fitness gains and broader repertoires must be assessed separately. New lineage records trace policies retained by VC, but rejected by an MSE observer on the same candidate pool, to later record-setting offspring (Appendix S). Together, these studies support survivor selection as a contributor to the gains and clarify its implications for learned-descriptor QD.

Manuscript Suggestions

Notation and QD-score definitions

We now define the observations oi,to_{i,t}, their relationship to trajectory τi\tau_i, encoder and decoder parameters ϕ\phi and ψ\psi, and prediction o^i,t(ϕ,ψ)\widehat{o}_{i,t}(\phi,\psi) around Equation 1. Section 5 defines coverage as the occupied-cell fraction and QD score as the sum of best fitness over occupied passive-archive cells, adding the task-specific offset once per occupied cell.

Narrative, figures, and supporting material

We thank the reviewer for their suggestions on narrative flow and figure presentation, which helped us connect the analyses more clearly and make the main findings easier to follow. We rewrote Section 6 around measured outcomes and Section 7 around their interpretation, with clearer connections between the studies. The original dense mechanism summaries now appear in Appendix O.3 (Figure 34 and Figure 35), with the blue control annotation outside the panels. We corrected the arrow and label spacing in Figure 1 and moved its shared colorbar outside the plotted data.

We recognize that the supporting material is extensive. The main text can be read independently: it introduces the idea of regularizing learned behavior spaces to shape archive selection, presents AURORA-VC and its main results, analyzes how the method affects search, and discusses its implications for QD. The appendix provides detailed results and descriptions of the studies summarized in the main text, along with additional figures.

State trajectories in the abstract

The abstract now refers to “observed state trajectories,” replacing “action trajectories.


We again thank Reviewer 6zsi for their detailed and constructive suggestions, which have helped strengthen the manuscript. We welcome any further questions and would be happy to provide additional clarification or supporting experimental results.

I thank the authors for the clarifications and the additional experiments. I think they improve the paper's presentation and contribution. I also have read the other reviews and the rest of the rebuttal. However, my assessment of the paper does not change much. I still find the paper unnecessarily difficult to follow. Much of the theoretical and mechanistic narrative could be simplified substantially without losing scientific content. In my view, the work does not yet reach the level of clarity and conceptual contribution I would expect for acceptance at ICLR. Maybe I'm wrong, but I honestly think that LLM usage in this work has done more harm than good, and that the paper would be more enjoyable if written in plainer, more direct english.

Reviewer_MbQCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes learning better-structured behavior descriptors for QD by adding variance and covariance regularization to the reconstruction loss used in AURORA and PGA-AURORA. These terms encourage variation across descriptor dimensions and discourage correlations between them. The paper studies how this changes which policies are kept and how it affects fitness and behavioral diversity.

Critical strengths
  • The method is simple and can be added to existing learned-descriptor QD methods.
  • The paper includes many ablations and controlled studies. The whitening and policy-selection experiments are particularly useful for understanding why the method helps.
Critical weaknesses
  • Maximum fitness does not paint a full picture of how two QD methods compare. The paper also reports QD score and coverage, but these use handcrafted descriptors. They show how well the methods cover predefined behavioral differences, while leaving unclear whether the learned descriptors capture additional useful variations. Demonstrating that the resulting policies perform better in a downstream application, such as adapting to changed conditions, would provide stronger evidence.
  • It remains unclear whether the additional variation in the descriptors captures differences that policies can reliably produce or differences caused by randomness in the environment. For example, different initial conditions could give the same policy different descriptors. The regularization encourages variation without distinguishing these sources. Evaluating the same policies across multiple episodes would help clarify this.
  • I found Sections 6 and 7 difficult to follow. There are many ablations, but I had trouble understanding what broader conclusions we should draw from them. For instance, Section 7.3 shows that policies stay longer and older parents are reused, while Section 7.5 tests whether the choice of policies matters for later progress. Connecting these studies more directly would help. I would prefer fewer separate diagnostics in the main text and more explanation of the important findings and their implications for QD.
  • The related work should discuss existing treatments of behavior-space geometry, including Density Descent, AutoQD, and work on centroid generation [1-3]. In particular, it would help to contrast covariance regularization with such approaches.

[1] Mouret, Fast Generation of Centroids for MAP-Elites, GECCO 2023.
[2] Hedayatian and Nikolaidis, AutoQD: Automatic Discovery of Diverse Behaviors with Quality-Diversity Optimization, ICLR 2026.
[3] Lee et al., Density Descent for Diversity Optimization, GECCO 2024.

Questions for authors
  1. How many independent episodes are used to estimate each policy's fitness and descriptor? Are initial conditions fixed, and are retained policies ever evaluated again?
  2. Do the gains come mainly from keeping policies that can produce better offspring, or from using their trajectories to train the next encoder? Could these two effects be tested separately?
  3. Can the authors show an example where VC keeps a type of policy that reconstruction alone would discard, explain what is different about its behavior, and trace how it helps produce a better policy later? Such qualitative analysis would strengthen the paper's argument.
3 comments
AuthorsOpenReview ↗

Response to Reviewer MbQC [1/2]

We thank Reviewer MbQC for their detailed comments and suggestions, which helped us broaden the evaluation through downstream adaptation and descriptor repeatability studies. Their questions also led us to deepen the mechanistic analysis by separating parent reuse from encoder-training data and tracing retained policies to later record-setting offspring. We address all points below and have revised the manuscript accordingly. Additions to the manuscript are marked in blue. Section numbers cited below refer to the revised manuscript.

We appreciate the reviewer's recognition of the method's simplicity and compatibility with existing learned-descriptor QD methods, and of the value of our ablations and controlled studies, particularly the whitening and policy-selection experiments.

Downstream Usefulness of the Archives

We thank the reviewer for highlighting the importance of evaluating useful behavioral variation beyond the predefined reporting dimensions. Adaptation to changed conditions provides a complementary test of repertoire usefulness (e.g. Cully et al., Robots that can adapt like animals (Nature, 2015)). We use fixed handcrafted reporting descriptors to compare runs whose learned behavior spaces differ, following prior learned-descriptor QD evaluations (Chalumeau et al., ICLR 2023). We examine both the performance available from existing policies and the fitness achieved when search continues under the new conditions.

We added frozen-archive transfer and continual-learning studies in Ant-Uni, HalfCheetah, and Walker, selected for their clear primary fitness gains (Section 6.4; Appendix Q).

In the frozen-archive study, policies, encoders, and normalization remain fixed, and policy selection and fitness evaluation use separate episodes. At three times the original gravity, HalfCheetah fitness is approximately 3,430 with VC versus 2,647 with MSE; at twice the original actuator strength, Walker reaches approximately 4,288 versus 1,679. The complete sweeps appear in Appendix Q.5 (Figure 40).

In the continual-learning study, search continues through sudden changes in gravity, friction, torso mass, or leg strength, with ten paired seeds per environment. Under sustained changes, normalized AUC of median maximum fitness, summarized across the four perturbations, is 7,120 versus 5,824 in Ant-Uni, 6,185 versus 5,766 in HalfCheetah, and 4,868 versus 2,674 in Walker (VC versus MSE; Appendix Q.4).

These results demonstrate the usefulness of VC archives under changed conditions and during continued learning. They highlight the value of assessing QD archives through the policies they make available and the subsequent search they support, alongside passive QD score and coverage.

Descriptor Repeatability and Evaluation Protocol

Section 5 and Appendix A.1 now clarify that, during primary search, each policy's fitness and descriptor use one episode from an initial state shared across policies and fixed within the run. This provides common starting conditions for comparing policies during search. Search reuses the recorded fitness and trajectories of retained policies, re-encoding the stored trajectories after encoder updates.

Our additional repeatability study assesses how the learned repertoires generalize across fresh native resets, complementing the fixed-state training protocol. We evaluated 128 sampled policies per archive over 100 episodes, across ten tasks and ten seeds per method, holding policies, encoders, and normalization fixed (Appendix T). Repeatability measures between-policy separation relative to within-policy episode variation. The estimates show differences across tasks and methods, with higher paired repeatability for MSE in HalfCheetah, Walker, and Humanoid-Uni. Each method is evaluated on its own archive, so these comparisons characterize the policies and their learned representations together. These evaluations add a measure of consistency across episodes alongside the fitness and downstream-usefulness results.

AuthorsOpenReview ↗

Response to Reviewer MbQC [2/2]

Separating Parent Reuse from Encoder-Training Data

We thank the reviewer for suggesting this experiment, which strengthened the manuscript by separately examining the roles of retained policies as parents and as sources of encoder-training data. Following the reviewer’s suggestion, we tested whether VC-selected policies improve later search through the offspring they produce, the trajectories they supply for encoder training, or both.

We ran a four-arm study independently assigning MSE- or VC-selected policy banks to parent sampling and encoder training: ten tasks x ten seeds x four arms, each for 5,000 updates (Appendix R). The four arms are M/M, V/M, M/V, and V/V, with the parent-bank assignment listed first and the encoder-training-bank assignment second; M and V indicate selection using MSE- and VC-trained descriptor distances, respectively. M/M is a controlled hybrid arm, not the native MSE baseline: both banks are MSE-selected, while active competition still uses VC distances. All four arms share this VC-based competition rule and MSE-based stored coordinates and rebuilds. We hold these rules fixed to compare the parent and training-data assignments within a common search configuration. Using the native MSE baseline as M/M would also switch the competition metric from VC to MSE, mixing that change with the effects of the bank assignments.

The study does not identify a consistently dominant route. All conditional-effect intervals include zero in Ant-Uni, HalfCheetah, and Walker. In Humanoid-Omni, VC-selected training data improve fitness when parents are also VC-selected, demonstrating a contribution through encoder training in this controlled configuration.

A Retained Policy's Behavior and Later Contribution

We instrumented 100 native VC runs across all ten tasks with an MSE observer trained on the VC archive's trajectories; it ranks the same candidate pools without controlling search (Section 7.3; Appendix S). This identifies policies VC retains that MSE geometry would reject, then traces their later offspring.

In a post-hoc HalfCheetah example, a policy ranks 978 under VC and 1,106 under MSE at the 1,024-policy cutoff. Its first-foot contact fraction is 0.195, compared with 0.369 for its nearest strictly fitter MSE neighbor. Its policy-gradient child one update later reaches fitness 4,518, improving the run record by approximately 298 (Appendix S.3). Across the cohort, 2,217 of 5,005 record-setting children have a direct parent retained after an earlier VC-keep/MSE-reject disagreement. These records connect a concrete behavioral difference and selection disagreement to realized reproductive use and subsequent progress.

Manuscript Suggestions

Narrative and implications for QD

We agree that the results needed a clearer interpretation in terms of QD search. The revised Section 6 distinguishes maximum fitness, passive repertoire quality, and downstream usefulness, while Section 7 explains how learned geometry changes competition, retention, and reuse. In particular, the survivor interventions show that changing which policies survive can improve later fitness even when admission counts are controlled, while coverage can decrease. The lineage examples trace retained policies to later record-setting offspring. These findings clarify the QD implication: learned descriptor geometry shapes which search opportunities an archive preserves, and higher fitness, broader coverage, and useful archive memory need to be assessed separately. The frozen-archive and continual-learning studies add complementary evidence of usefulness under changed conditions (Section 6.4).

Related work on behavior-space geometry

We thank the reviewer for recommending work on centroid generation, Density Descent, and AutoQD. These suggestions helped us broaden Section 2 and clarify how our regularization relates to other approaches to behavior-space geometry. We contrast arranging archive reference points, guiding exploration through feature-density estimates, and constructing occupancy-based descriptors with directly regularizing the trajectory encoder's variance and covariance during reconstruction training. Our encoder objective uses no fitness targets.


We again thank Reviewer MbQC for their detailed and constructive suggestions, which have helped strengthen the manuscript. We welcome any further questions and would be happy to provide additional clarification or supporting experimental results.

Thank you for your detailed responses, and apologies for my late reply.

I appreciate the authors additions and explanations, particularly in sections 6 and 7 of the manuscript which I believe have improved substantially. I maintain my positive assessment of the work and I'll keep an eye on the discussions with other reviewers as well.

Reviewer_13c7OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors design a new algorithm based on geometric regularisation of learned behavior spaces for unsupervised QD. The paper provides extensive comparisons and metrics. The algorithm designs an AURORA like algorithm based on auto-encoder but add a geometric regularisation with a penalty of covariance and variance to the reconstruction. The penalty is based on VIC Reg; VIC Reg is based on a variance term, (γ−∑ivi)(\gamma - \sum_i v_i), which forces within-batch latent samples to be different (preventing collapse) and a covariance term, 1d∑r≠j(Crj)2\frac{1}{d} \sum_{r\neq j} (C_{rj})^2, which forces the decorrelation of the different dimensions. This way, it encourages variation across descriptor dimensions and discourages correlations between the distinct dimensions. The main improvement seems to be in the maximal fitness rather than the diversity. Survival-selection interventions are used to confirm that selection is one of the sources of the improvement.

Critical strengths
  • Well-reasoned algorithm and exposition
  • Extensive benchmark set.
  • Statistically sound execution
Critical weaknesses
  • There seems to be only a benefit in the maximal fitness and not in the QD score. Unfortunately, this is a significant limitation in the context of QD algorithms.
  • The experiments are only in robotics but this is a minor weakness only.
Questions for authors

Abstract:

  • the abstract uses the term VC but does not introduce it yet.
  • The last sentence should then probably say “VC regularization” or “geometric regularization” instead of “VC”
  • what kind of selection? The VC penalty adds a penalty to the loss used to define the BD. So how does it interact with the selection? Which kind of selection rule is used for the QD algorithm and how do the selection rule and the VC penalty interact? A bit more information in the abstract would be helpful.#

Eq. 1:

  • o_i is undefined. Presumably these are observations.
  • The ϕ\phi and ψ\psi are not used in the equation

B.2.1:
"A diagonal covariance with very unequal variances can have zero V+C loss." Why is this? I would expect the V part to contribute to the loss whenever the variance is lower than the threshold; this is technically consistent with the statement but there is basically a lower bound on the allowed variance for the expression γ−V\gamma - V to be zero.

“In this experiment, we investigate whether VC can improve search through its survivor rule while the active archive continues to store and use MSE descriptors.” What is meant by MSE descriptors? Probably you mean descriptors updated from MSE loss function? It should be clarified.

Appendix A.2: “Pool incumbents and candidates into P. “ The sentence flows strangely in conjunction with the previous one. Also not clear what is P at this point.

Appendix H: the difference of the adaptive-radius archive to the other algorithms should be clarified.

Appendix P.2: wouldn’t the two-by-two factorial design always have four arms?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 13c7 [1/2]

We thank Reviewer 13c7 for their detailed comments and suggestions, which helped us clarify the limitations of our QD claims, expand the evaluation with continual learning results, and improve the manuscript’s clarity and presentation. Their questions about the variance threshold, equation notation, and archive selection also helped us make the mathematical conditions and algorithm descriptions more precise. We address all points below and have revised the manuscript accordingly. Additions to the manuscript are marked in blue. Section numbers cited below refer to the revised manuscript.

We appreciate the reviewer’s positive assessment of our algorithm and presentation, the breadth of our benchmarks, and the statistical soundness of our evaluation. We agree that the focus on simulated robotics is a limitation of our paper. Within this scope, we aim to maintain task diversity through ten distinct tasks spanning different robot morphologies, locomotion objectives, and navigation challenges, including two-dimensional maze solving.

Benefits Beyond Maximum Fitness

We agree that VC improves passive QD score less consistently than it improves maximum fitness, and we now make this distinction explicit in our manuscript. Across the 20 task–algorithm settings, passive QD score significantly favors VC in eight and MSE in three (Table 1).

As noted by Reviewer MbQC, QD score and coverage are less straightforward to assess, as they are evaluated relative to an a priori fixed set of behavior descriptors rather than the dimensions of variation learned within each individual run. Thus, another way we could think about evaluating an archive's quality is to ask how well it can adapt to future variation (e.g. Cully et al., Robots that can adapt like animals (Nature, 2015)). While this is again an indirect proxy for QD score or coverage, it provides additional evidence to complement these measures.

In the revised manuscript, we evaluate this via new sections on continual learning and frozen-archive transfer in Ant-Uni, HalfCheetah, and Walker, selected for their clear primary fitness gains (Section 6.4; Appendix Q).

In the frozen-archive study, we test existing policies under changed robot physics without further learning, selecting policies and measuring their fitness on separate episodes. VC archives provide higher-fitness policies across many perturbation settings: at three times the original gravity, HalfCheetah fitness is approximately 3,430 versus 2,647 for MSE; at twice the original actuator strength, Walker fitness is approximately 4,288 versus 1,679. The full results are given in Appendix Q.5 (Figure 40).

In the continual-learning study, search continues through sudden changes in gravity, friction, torso mass, or leg strength, with ten paired training seeds per environment. Under sustained changes, normalized AUC of median maximum fitness, summarized across the four perturbations, is 7,120 versus 5,824 in Ant-Uni, 6,185 versus 5,766 in HalfCheetah, and 4,868 versus 2,674 in Walker (VC versus MSE).

These results show that VC's lower passive QD score in HalfCheetah need not translate into poorer performance under changed conditions. Together, these findings highlight the value of assessing QD archives through the policies they make available and the subsequent search they support, alongside passive QD score.

AuthorsOpenReview ↗

Response to Reviewer 13c7 [2/2]

Manuscript Suggestions

Suggestions on the abstract

We thank the reviewer for these suggestions on the abstract, which helped us define VC and explain how the learned descriptors enter archive selection. We have incorporated them into the abstract in the revised manuscript.

Corrections to Equation 1

We now define τi=(oi,1,…,oi,T)\tau_i=(o_{i,1},\ldots,o_{i,T}), with oi,to_{i,t} the normalized observation vector at sampled step tt of trajectory ii. We also define ϕ\phi and ψ\psi as encoder and decoder parameters and explicitly write the prediction as o^i,t(ϕ,ψ)\widehat{o}_{i,t}(\phi,\psi).

Zero V+C loss with unequal variances

We thank the reviewer for highlighting the variance threshold, which helped us state the conditions for zero V+C loss more precisely. Appendix B.2.1 now explicitly requires every coordinate variance to be at least γ2−ϵ\gamma^2-\epsilon for the variance term to vanish (Equation 9). Zero covariance loss additionally requires zero off-diagonal covariance. There is no upper variance bound: for example, diagonal variances 1 and 100 satisfy both terms under our defaults.

Meaning of “MSE descriptors”

Appendix P.3 now specifies “descriptors from an encoder trained with reconstruction MSE alone,” identifying the objective used to learn those coordinates.

Definition of the pool PP

Appendix A.2 now introduces PP as the combined pool of current archive policies and newly proposed candidates before defining the selection rule.

Adaptive-radius archive versus the other algorithms

Appendix H now distinguishes global DNS priority ranking at fixed capacity from local radius conflicts and a fitness-ordered rebuild with occupancy control. PGA specifies candidate generation; MSE and MSE+VC specify encoder training. Both objectives use the same adaptive-radius selector, whose custom admission, rebuild, and radius-rescaling rules are stated explicitly.

Four arms in a two-by-two factorial

Yes. We removed the redundant description and instead name the two factors: the metric for neighbor ordering and the metric for distance scoring, each using MSE or VC geometry. All four combinations remain (Appendix P.2).


We again thank Reviewer 13c7 for their detailed and constructive suggestions, which have helped strengthen the manuscript. We welcome any further questions and would be happy to provide additional clarification or supporting experimental results.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Closing Author Statement

We thank the reviewers for their constructive feedback and the Area Chair for considering the reviews, discussion, and revised manuscript.

Our central finding is that learned descriptor geometry shapes which policies an archive preserves for future search. Reconstruction alone does not uniquely determine the distances between behaviors, yet these distances govern archive competition and survival. By shaping this geometry, VC regularization can change which policies remain available for later discoveries. Our contribution is to investigate these consequences for online QD search, rather than simply applying an existing regularizer to a new setting.


During the rebuttal period, we addressed the reviewers' concerns through substantial revisions and additional experiments:

  • Downstream usefulness and benefits for QD: We added frozen-archive transfer and continual-learning experiments under changed robot physics in three environments. These evaluations provide evidence that VC archives retain useful policies and support continued search under conditions beyond the original training environment.

  • How retained policies contribute to future search: We added lineage analyses identifying policies retained by VC but rejected by an MSE observer evaluating the same candidate pools. Across 100 runs spanning ten tasks, 44% of record-setting offspring (2,217 of 5,005) had a direct parent previously involved in such a selection disagreement, with examples in 99 of 100 runs. These observations complement our controlled survivor-selection interventions, which demonstrate that survivor identities can affect subsequent search outcomes. We also added a four-arm study separating retained policies' roles as parents and sources of encoder-training data, finding no consistently dominant mechanism across tasks.

  • Reliability of learned descriptors: We added repeated evaluations across fresh initial states on all ten tasks, documenting task-dependent repeatability.

  • Clarity and scope: We clarified the mathematical motivation, notation, relative effect sizes, and variance/covariance component comparisons. We also reorganized Sections 6 and 7 to distinguish maximum fitness, repertoire quality, downstream usefulness, and the mechanisms underlying search improvements.

Two reviewers have responded to our rebuttal so far. Reviewer MbQC stated that Sections 6 and 7 had improved substantially and maintained their positive assessment. Reviewer 6zSi also acknowledged improvements to presentation and contribution, but retained concerns about clarity and conceptual significance. We take these concerns seriously and will further simplify the narrative in the final manuscript.


We believe the main contribution is demonstrating that behavior-space geometry is consequential for QD search, not merely for representation learning. Our experiments connect changes in learned distances to archive selection, policy retention, subsequent fitness improvements, and usefulness under changed conditions. These fitness gains are accompanied by task-dependent effects on QD score, coverage, and descriptor repeatability. We hope this work helps motivate further research into learning behavior spaces that preserve opportunities for future discovery and adaptation.

We thank the reviewers and Area Chair again for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.