- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies Quality Diversity (QD) methods mainly in continuous control environments, and in the setting of neuroevolution. It follows a line of work aiming at learning a diverse collection of neural policies (called an archive), the diversity criterion being learned automatically instead of being human-specified like in ordinary QD methods. Here the diversity criterion comes from the distance between latent vectors of an auto-encoder trained to reconstruct the trajectories of the policies, the main instance of this family of algorithm is AURORA that combines MAP-elites search with the reconstruction-based behavioral descriptors. The main contribution of the work is to identify a failure mode of AURORA (low reconstruction loss doesn't guarantee diversity) and adding two regularization terms, variance-covariance (VC), inspired from self-supervised learning in order to improve the geometry of the latent space. The paper then reports improved performances in terms of max fitness and qd-score (sum of fitness of the candidates in the archive) on a benchmark composed of 10 games, the reported improvement is not uniform on all tasks, and is not a huge performance improvement. Finally the paper also tries to uncover the mechanisms explaining the advantage of their regularization.
Critical strengths
None
Critical weaknesses
Presentation
I think this should be a fairly simple paper to read, and I find the phrasing and presentation obscure, and difficult to parse, even though I'm pretty familiar with the field and related works.
-
Notations: they are not well-defined, take Equation 1 alone, are not defined and the link with the mentioned in the paragraph above is not explicit. QD score is never defined, which would make it hard for the broader ICLR community to understand the paper.
-
Maths: I don't find Proposition 1 self-explanatory, I guess it's a counter-example of the "good reconstruction makes latent codes separable" statement, but I think it deserves a more pedagogical treatment. I don't even think that statement is assumed in the community, so I don't really see the point of trying to disprove it.
-
Phrasing and narrative flow: I think the writing needs more polishing. I find the phrasing overly-complex regarding the matter of study. And I think the (acknowledged) use of LLM in writing did not help.
-
Presentation of the results: I find the presentation of the results hard to follow. In the main text, I think the relative improvement of using the VC regularization compared to vanilla AURORA/PGA-AURORA is not easy to assess in Table 1. I also think ablation studies w/o C, w/o V should be in the main text and clearly highlighted. I couldn't follow well the mechanistic analysis of why/how VC helps, Figures 6 and 7 are very information-dense with a lot of metrics defined along the way. Moreover, I think 80 pages of Appendix is a huge amount of material for reviewers. Appendix B contains a lot of theoretical results, that I haven't checked, but it is not clear to me whether or not they belong in this paper, as they are only briefly mentioned in the main text.
Significance:
-
Significance of the method: I think this work is mainly incremental, take AURORA/PGA-AURORA, add a regularization and see if/why it helps. It is not rejecting grounds in itself, but it calls for a paper with a better presentation and clearer advantage compared to previous works.
-
Significance of the evaluation: I don't think the benchmark used is very significant, I know it's been used in prior works, but it kind of fails to demonstrate performance on the tasks QD actually cares about: using the archive for robustness, hierarchical RL or any interest task. I think the broader community of ICLR would be more interested in these downstream metrics than QD-score/fitness alone, that are only proxy measures in the end.
Minor
- In the abstract you say:
Prior works, like AURORA, train autoencoders on action trajectories to find these important dimensions of variation.
If I'm not mistaken, you probably meant state trajectories.
- Overlapping Text/arrows in Figure 1 and 6.
Questions for authors
I don't think the rebuttal could significantly change my assessment. I will still ask clarifying questions:
-
Can you clarify the demonstration made in Section 3, through Proposition 1 and Figure 1?
-
Can you clarify the importance of the mathematical analysis in Appendix B? Is it a standalone contribution or a motivation for the proposed regularization?
-
I think providing relative improvement (in %) w.r.t Aurora could make the advantage of the VC regularization clearer.
3 comments
Response to Reviewer 6zsi [2/2]
Magnitude of the Gains and Component Ablations
Section 6.1 now reports relative improvements directly. In Walker, median maximum fitness increases from approximately 3,850 to 6,510 with PGA-AURORA (+69%) and from 1,660 to 2,230 with AURORA (+34%). The corresponding Ant-Uni gains are approximately 19% and 24%. We also make the task dependence explicit: Ant-Omni gains are small near its fitness ceiling, and HalfCheetah's approximately 4% maximum-fitness gain with PGA accompanies an approximately 11% lower passive QD score.
We brought the component comparison into Section 6.2. Walker fitness medians are approximately 3,850 with MSE, 2,830 with V only, 3,740 with C only, and 6,510 with V+C. Neither component alone reproduces the combined gain there or in Ant-Uni; the full task-level comparison appears in Table 9 (Appendix L).
Contribution and Implications for QD
We explicitly attribute the variance and covariance penalties to VICReg. Our contribution is to investigate how conditioning an online learned behavior space changes archive selection and the policies available for future search. We test this with both genetic and policy-gradient proposals and with a second archive-selection rule beyond DNS.
The revised Section 7 connects changes in competition to policy retention, reuse, and controlled interventions on survivors. Changing survivor identities can improve later fitness while reducing coverage, showing why fitness gains and broader repertoires must be assessed separately. New lineage records trace policies retained by VC, but rejected by an MSE observer on the same candidate pool, to later record-setting offspring (Appendix S). Together, these studies support survivor selection as a contributor to the gains and clarify its implications for learned-descriptor QD.
Manuscript Suggestions
Notation and QD-score definitions
We now define the observations , their relationship to trajectory , encoder and decoder parameters and , and prediction around Equation 1. Section 5 defines coverage as the occupied-cell fraction and QD score as the sum of best fitness over occupied passive-archive cells, adding the task-specific offset once per occupied cell.
Narrative, figures, and supporting material
We thank the reviewer for their suggestions on narrative flow and figure presentation, which helped us connect the analyses more clearly and make the main findings easier to follow. We rewrote Section 6 around measured outcomes and Section 7 around their interpretation, with clearer connections between the studies. The original dense mechanism summaries now appear in Appendix O.3 (Figure 34 and Figure 35), with the blue control annotation outside the panels. We corrected the arrow and label spacing in Figure 1 and moved its shared colorbar outside the plotted data.
We recognize that the supporting material is extensive. The main text can be read independently: it introduces the idea of regularizing learned behavior spaces to shape archive selection, presents AURORA-VC and its main results, analyzes how the method affects search, and discusses its implications for QD. The appendix provides detailed results and descriptions of the studies summarized in the main text, along with additional figures.
State trajectories in the abstract
The abstract now refers to “observed state trajectories,” replacing “action trajectories.
We again thank Reviewer 6zsi for their detailed and constructive suggestions, which have helped strengthen the manuscript. We welcome any further questions and would be happy to provide additional clarification or supporting experimental results.
I thank the authors for the clarifications and the additional experiments. I think they improve the paper's presentation and contribution. I also have read the other reviews and the rest of the rebuttal. However, my assessment of the paper does not change much. I still find the paper unnecessarily difficult to follow. Much of the theoretical and mechanistic narrative could be simplified substantially without losing scientific content. In my view, the work does not yet reach the level of clarity and conceptual contribution I would expect for acceptance at ICLR. Maybe I'm wrong, but I honestly think that LLM usage in this work has done more harm than good, and that the paper would be more enjoyable if written in plainer, more direct english.
Response to Reviewer 6zsi [1/2]
We thank Reviewer 6zsi for their detailed comments and suggestions. Their requests for clearer mathematical motivation, relative improvements, and component ablations helped us make the main contribution and its empirical benefits easier to assess. Their emphasis on downstream applications also led us to expand the evaluation with frozen-archive transfer and continual-learning studies. We address all points below and have revised the manuscript accordingly. Additions to the manuscript are marked in blue. Section numbers cited below refer to the revised manuscript.
We acknowledge that the motivation, mathematical analysis, and experimental findings needed a clearer presentation. We have rewritten the main narrative, made effect sizes and component comparisons explicit, and added downstream evaluations of the archives.
Downstream Usefulness of the Archives
We thank the reviewer for emphasizing the practical uses of QD archives, including robustness and hierarchical control. We have expanded our evaluation to examine robustness under changed operating conditions, connecting the benchmark results to the usefulness of the discovered repertoires. A repertoire can provide alternative policies when the original conditions change, as demonstrated for damage recovery by Cully et al., Robots that can adapt like animals (Nature, 2015). This motivates evaluating both the competence already available in the archive and its value for further learning.
We added frozen-archive transfer and continual-learning studies in Ant-Uni, HalfCheetah, and Walker, selected for their clear primary fitness gains (Section 6.4; Appendix Q). Each study uses ten training seeds per method and environment.
Frozen probes evaluate existing policies under changed physics without further learning, using separate episodes for policy selection and fitness evaluation. At three times the original gravity, HalfCheetah fitness is approximately 3,430 with VC versus 2,647 with MSE; at twice the original actuator strength, Walker reaches approximately 4,288 versus 1,679. The complete sweeps appear in Appendix Q.5 (Figure 40).
To evaluate how the repertoires support further search, we continued learning through sudden changes in gravity, friction, torso mass, or leg strength. Under sustained changes, normalized AUC of median maximum fitness, summarized across four perturbations, is 7,120 versus 5,824 in Ant-Uni, 6,185 versus 5,766 in HalfCheetah, and 4,868 versus 2,674 in Walker (VC versus MSE; Appendix Q.4).
Together, these results connect the benchmark gains to effective policies under altered physics and higher post-change fitness AUC during continued learning. These downstream tests provide complementary evidence of the practical value of the learned repertoires alongside QD score and coverage.
Reconstruction and Archive Selection
We have clarified the purpose of Proposition 1 in Section 3: reconstruction alone leaves freedom in the distances used for archive selection. For example, doubling one latent coordinate and undoing that scaling before decoding preserves reconstruction, but can change nearest neighbors and which policies survive. The result identifies a freedom left by the reconstruction objective; it does not assume that reconstruction guarantees behavioral separability.
Figure 1 demonstrates the search consequence using a fixed pool of 8,192 PointMaze trajectories. The illustrated transformation changes 241 of the 1,024 selected policies while preserving the policies, trajectories, fitnesses, and reconstruction up to numerical error. Thus, equally accurate reconstructions can induce different archive decisions.
Appendix B provides the derivations and experimental details supporting the PointMaze study, making explicit how descriptor geometry can change archive membership while reconstruction remains fixed. It also develops supporting analysis of VC regularization, archive competition, and parent reuse to clarify the scope of our mechanistic interpretation. Section 3 now explains the role of this supporting analysis.