acceptodds
Under review as a conference paper at ICLR 2027

Differentiable Quality Diversity with Cross-Task Collaborative Discovery

Abstract

Quality-Diversity (QD) optimization aims to discover collections of high-performing solutions with diverse behaviors. In many applications, however, multiple related QD tasks may need to be solved due to different execution environments. Existing methods typically optimize such multiple tasks independently, without explicitly coordinating their search processes, which can lead to repeated exploration of related regions and redundant rediscovery of useful solutions. In this paper, we propose Multitask Differentiable Quality-Diversity with Cross-Task Collaborative Discovery (MTQD-CoD), which jointly optimizes multiple related QD tasks through cross-task collaboration. Specifically, a subspace-projected repulsion mechanism encourages task-specific searchers to explore less-overlapping regions, while an inter-task transfer probability adaptation mechanism adjusts cross-task transfer probabilities according to the observed archive contributions of transferred candidates, thereby regulating how strongly discoveries are shared across tasks. By reducing redundant exploration while reusing transferable discoveries, MTQD-CoD creates more opportunities to explore undercovered behavioral regions and identify additional high-quality solutions. Experiments on arm repertoire, latent-space illumination, and policy optimization problems demonstrate the effectiveness of MTQD-CoD in discovering diverse and high-quality solutions across multiple related QD tasks.

Primary area:
optimization
Keywords:
Multitask Optimization, Quality-Diversity Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FceSOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a multitask differentiable quality-diversity optimization framework that encourages complementary exploration and adaptive knowledge transfer across related tasks. The motivation is intuitive, and the method is relatively modular, combining subspace-projected repulsion with adaptive inter-task transfer probabilities. Experiments on arm repertoire, latent-space illumination, and policy optimization tasks demonstrate improvements over the corresponding single-task baselines.

Critical strengths

The motivation is intuitive and easy to follow. Reducing redundant exploration while sharing useful discoveries across related tasks is a reasonable objective. The proposed framework also provides a relatively straightforward way to incorporate these ideas into existing differentiable QD algorithms.

Critical weaknesses
  1. Limited clarity on novelty. While I am not very familiar with the multitask QD literature, the central idea of reducing redundant exploration and transferring useful solutions across related tasks seems relatively natural in multitask optimization. I am not fully convinced that the paper sufficiently distinguishes its contributions from existing multitask QD or evolutionary multitasking approaches.

  2. Limited performance gains over strong baselines. On the Arm Repertoire and LSI benchmarks, the improvements over CMA-MAEGA are generally modest. Although the gains are more substantial in policy optimization, MTQD-CoD still underperforms DCG-ME on Ant. It is unclear whether the improvements justify the additional computational overhead.

  3. Insufficient evidence for the proposed mechanisms. It is not entirely clear whether the proposed repulsion mechanism actually reduces redundant exploration, or whether adaptive transfer provides substantial benefits over simpler fixed-probability transfer strategies. The effectiveness of these mechanisms when tasks are weakly related or conflicting also remains unclear. These issues are important for understanding whether both components are necessary.

Questions for authors
  1. Could the authors clarify the novelty of MTQD-CoD relative to existing multitask QD and evolutionary multitasking methods?

  2. What is the additional computational overhead of MTQD-CoD, and how does it compare with the observed performance gains?

  3. Can the authors provide more direct evidence that repulsion reduces search overlap and that adaptive transfer outperforms fixed transfer probabilities, particularly for weakly related tasks?

9 comments
AuthorsOpenReview ↗

Response — (1/4)

Response to W1 and Q1

We thank the reviewer for the constructive comments. We would like to clarify that explicitly reducing redundant exploration across tasks is not common in conventional multitask optimization or multitask QD. In fact, our approach is somewhat counterintuitive compared with conventional multitask optimization, as conventional multitask optimization methods (in most cases) commonly encourage different tasks to explore similar regions rather than different ones (we will explain why this happened in below). We believe this represents a key conceptual novelty of MTQD-CoD, which introduces a “Repulsion + Transfer” strategy instead of the more conventional multitask optimization's “Attraction + Transfer” paradigm. To further clarify this distinction, we have also added a simple illustrative cartoon to the revised manuscript to help readers better understand the difference between the conventional multitask optimization strategy and our proposed “Repulsion + Transfer” strategy (see Fig. 7 on page 18). Furthermore, to the best of our knowledge, MTQD-CoD maybe the first attemption specifically designed for differentiable settings (in multitask QD), an area that remains unexplored. Existing multitask QD methods primarily focus on black-box optimization, whereas our approach explicitly exploits gradient information to enable complementary exploration and adaptive knowledge transfer.

Why is “Repulsion + Transfer” different from conventional multitask optimization?

The key difference lies in the search purpose. Conventional multitask optimization primarily aims to find high-quality solutions more efficiently. When tasks objective landscape are (positive) correlated, encouraging them to search similar promising regions can facilitate the improvement on the convergence, since promising regions searched by one task may also guide another one to identify high-quality solutions. This motivates common multitask optimization mechanisms such as genetic exchange and distribution alignment. Existing multitask QD methods also largely emphasize knowledge sharing and alignment. For example, Hou et al. (2025) [r1] employ VAE-based modeling and alignment to facilitate cross-task transfer. However, QD optimization aims to discover many diverse, high-quality solutions rather than only a few high-quality ones. If multiple tasks repeatedly explore the same promising regions, they may rediscover similar solutions, resulting in redundant search efforts. This motivates our alternative strategy, i.e., encouraging different tasks to explore complementary regions while still sharing their useful discoveries

[r1] Hou, Yaqing, et al. "Evolutionary heterogeneous multitasking for quality diversity optimization." IEEE Transactions on Evolutionary Computation (2025).

AuthorsOpenReview ↗

Response — (2/4)

Response to W2 and Q2 (Part 1)

Thank you for your comments. To further demonstrate the effectiveness of MTQD-CoD and clarify when cross-task collaboration provides greater benefits, we conducted additional experiments on Arm and LSI. The new results show more improvements over CMA-MAEGA, particularly in the four-task LSI setting, where MTQD-CoD consistently achieves better Coverage and Normalized QD Scores across all tasks. We also provide a computational complexity introduction and wall-clock measurements to evaluate the additional overhead of the proposed mechanisms.

(1) Performance improvements on Arm

The relatively small improvements on Arm in the previous manuscript are partly due to performance saturation. These problems are relatively easy for CMA-MAEGA, whose CMA-ES-based gradient emitter already provides strong exploration capabilities, leaving limited room for additional improvement. To further investigate this, we conducted a new experiment by reducing the initial CMA-ES step size from σ0=0.05\sigma_0=0.05 to σ0=0.005\sigma_0=0.005 for both CMA-MAEGA and MTQD-CoD (MAE). This allows us to examine whether cross-task collaboration becomes more beneficial when the exploration capability of the independent optimizer is more constrained.

CMA-MAEGAMTQD-CoD (MAE)
TaskCoverageNorm. QD ScoreCoverageNorm. QD Score
Arm-1 / Task-173.18% ± 1.09%3.66 ± 0.0574.58% ± 0.94%3.73 ± 0.05
Arm-1 / Task-273.47% ± 1.45%3.67 ± 0.0774.52% ± 1.04%3.73 ± 0.05
Arm-2 / Task-137.90% ± 0.58%1.89 ± 0.0338.74% ± 0.63%1.94 ± 0.03
Arm-2 / Task-238.07% ± 0.54%1.90 ± 0.0338.40% ± 0.68%1.92 ± 0.03
Arm-3 / Task-160.39% ± 0.85%3.02 ± 0.0461.31% ± 1.07%3.07 ± 0.05
Arm-3 / Task-261.16% ± 0.83%3.03 ± 0.0461.27% ± 0.97%3.07 ± 0.05

The new results show that MTQD-CoD outperforms CMA-MAEGA in both Coverage and Normalized QD Score on all six Arm tasks, with larger improvements than those observed under the original configuration. This suggests that cross-task collaboration provides greater benefits when independent exploration is more limited.

(2) Performance improvements on LSI.

As discussed in Section III, the main benefits of MTQD-CoD come from reducing redundant exploration when different tasks have overlapping search regions or high-quality solution sets. However, the two tasks in some LSI settings differ considerably in their objectives and behavior descriptors, making such overlap less likely. This is also consistent with the observed dynamics of cross-task transfer probabilities in Fig. 19 which suggest relatively limited task relatedness in these settings. Consequently, there may be fewer opportunities for complementary exploration and useful knowledge transfer, which may explain the relatively modest improvements observed in the original experiments. A natural way to increase the opportunities for collaboration is to jointly optimize more tasks, since a larger task set may increase the chance that some tasks share overlapping search regions or useful discoveries. Motivated by this, we conducted an additional experiment that jointly optimizes four LSI tasks to investigate whether broader cross-task collaboration can lead to more substantial improvements. We therefore extended the experiments to jointly optimize four LSI tasks and compared MTQD-CoD (MAE, 4 Tasks) against independent CMA-MAEGA and the original two-task MTQD-CoD.

AuthorsOpenReview ↗

Response — (3/4)

Response to W2 and Q2 (Part 2)

TaskCMA-MAEGA / CoverageCMA-MAEGA / Norm. QDMTQD-CoD (MAE, 2 tasks) / CoverageMTQD-CoD (MAE, 2 tasks) / Norm. QDMTQD-CoD (MAE, 4 tasks) / CoverageMTQD-CoD (MAE, 4 tasks) / Norm. QD
LSI-1/Task-151.64% ± 1.65%33.20 ± 1.0453.75% ± 3.18%34.17 ± 1.9554.11% ± 2.05%34.48 ± 1.26
LSI-1/Task-280.03% ± 2.97%52.09 ± 1.8881.86% ± 2.77%52.75 ± 1.9984.14% ± 2.30%54.09 ± 1.50
LSI-2/Task-247.75% ± 2.82%30.12 ± 1.7450.59% ± 1.87%31.62 ± 1.1752.25% ± 1.11%32.73 ± 0.72
LSI-3/Task-278.96% ± 1.93%50.51 ± 1.2780.09% ± 3.33%50.76 ± 2.0983.52% ± 1.61%52.56 ± 0.89

As shown in the new results, the four-task MTQD-CoD achieves higher Coverage and Normalized QD Score than both baselines on all four tasks. For example, on LSI-3/Task 2, Coverage increases from 78.96% with CMA-MAEGA to 83.52%, while Normalized QD Score increases from 50.51 to 52.56. The improvements are also more obvious than in the original two-task setting. We have further added the corresponding convergence curves in Appendix J, Figure 20 (page 32) to show how these improvements develop throughout optimization. These results show higher performance gains than the strong baselines.

(3) Performance on Ant.

Regarding policy optimization, MTQD-CoD is developed by extending PPGA to the multitask setting and therefore inevitably inherits some of the limitations of its underlying QD optimizer. Ant presents a more challenging exploration setting, with a four-dimensional behavior space defined by the contact frequencies of its four legs. More importantly, Batra et al. (2024, Section 5, p. 9) explicitly acknowledge that PPGA's adaptive action standard deviation can impair exploration on Ant as PPO may reduce policy variance during training, limiting the discovery of new archive cells. This inherited limitation may partly explain why both PPGA and MTQD-CoD perform less competitively than DCG-ME on Ant. Nevertheless, MTQD-CoD consistently improves upon its PPGA backbone in mean Normalized QD Score on both Ant tasks demonstrating that cross-task collaborative discovery can provide additional benefits even when the underlying QD optimizer has limitations on a particular benchmark.

[r2] Batra, Sumeet, et al. "Proximal policy gradient arborescence for quality diversity reinforcement learning." International Conference on Learning Representations. Vol. 2024. 2024.

(4) Computational overhead

To evaluate the additional computational cost of MTQD-CoD, we provide both a complexity analysis and new wall-clock measurements. In terms of subspace-projected repulsion, although the decision space can be high-dimensional (e.g., 7,168 dimensions in LSI), subspace-projected repulsion operates in a much smaller gradient-coefficient space of dimension K+1K+1, where KK is the behavior descriptor dimension. In our experiments, this dimension is only 3–5. The additional operations mainly involve low-dimensional covariance projections and Mahalanobis distance calculations, with a computational complexity of

O(N2D(K+1)2+N2B(K+1)2+N2(K+1)3),O\left(N^2D(K+1)^2+N^2B(K+1)^2+N^2(K+1)^3\right),

where NN, DD, and BB denote the number of tasks, decision-space dimensionality, and batch size, respectively. In terms of adaptive transfer, it only requires categorical sampling and updates to an N×NN\times N transfer probability matrix. Moreover, candidates are reassigned across tasks rather than generating additional candidates, so the total evaluation budget remains unchanged. We further measured the running time of both mechanisms on Arm and LSI. The results are shown below.

BenchmarkIterationsTotal runtime (s)Subspace Projected Repulsion (s)Adaptive Transfer (s)
Arm (A1)20009.980.8190.669
LSI (P3)1000306.121.9620.563

These measurements allow us to assess the additional computational cost introduced by cross-task repulsion and adaptive transfer alongside the observed performance improvements.

Reviewer_pL1pOpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies multitask differentiable QD, where several QD problems share a solution space. The proposed MTQD-CoD algorithm encourages task-specific searches to explore different regions and adaptively assigns their candidate solutions to other tasks for evaluation.
Overall, the paper presents an interesting algorithm, but the gains over baselines are often modest. The policy-optimization results are somewhat promising, but their variability and the evaluation setup make the size and reliability of the benefits harder to assess.

Critical strengths
  • The paper is well written and clearly organized. Apart from a few small typos, I found it easy and enjoyable to read.
  • The figures and idealized examples do a good job providing intuition about the method and why coordinating searches could help.
Critical weaknesses
  • I would like a clearer motivation for solving several QD problems together. Concrete examples would help explain when separate task-specific archives are needed and why a multi-objective QD formulation would be inadequate.
  • The improvements on Arm and LSI are generally small, for instance when comparing CMA-MAEGA and MTQD-CoD (ME) relative to CMA-MEGA. Saturation may partly explain this, and if so results under smaller evaluation budgets would help establish whether collaboration offers a meaningful advantage earlier in search. The policy-optimization gains are larger, but also have substantially higher variance, particularly in Humanoid.
  • All experiments contain only two tasks, leaving scalability unclear. In addition, Arm-1 appears to contain identical optimization problems: scaling all link lengths cancels after normalizing the end-effector coordinates, and the objective remains unchanged. Its results therefore need to be distinguished from evidence of transfer between different tasks.
  • The evidence for improved sample efficiency could be clearer. Figure 6 compares only PPGA and MTQD-CoD using iterations. Including QD-PG and DCG-ME on a shared axis of environment steps or training time would make the efficiency comparison more informative. In the LSI domain, aside from LSI-2, MTQD-CoD provides little improvement to convergence speed or behavior (Figure 4).
Questions for authors
  1. What properties of a collection of tasks indicate that MTQD-CoD should substantially outperform independent searches? Where should we expect large gains over strong methods such as CMA-MAEGA?
  2. When archive improvements become rare, could the independence bias suppress useful transfer too early? Once a transfer probability becomes very small, can it recover if sharing becomes useful later?
  3. Could the authors explain how their repulsion mechanism relates to SQUAD [1], particularly the consequences of discouraging overlap between search distributions rather than between solutions in behavior space?
  4. What accounts for the large variation on Humanoid? Showing the individual runs would help clarify how consistently the method improves over PPGA.

[1] Hedayatian and Nikolaidis, Soft Quality-Diversity Optimization, ICLR 2026.

16 comments
AuthorsOpenReview ↗

Response — (1/6)

Response to W1.

We thank the reviewer for raising this important distinction. The key difference is that multi-objective QD optimizes multiple quality objectives within a same behavior space, whereas multitask QD jointly solves multiple QD problems, each with its own quality objective(s) and behavior space (i.e., behavior space or mapping can be different). Thus, multitask QD is not simply multi-objective QD with additional objectives.

The need for task-specific archives is particularly useful when tasks have different behavior descriptors. For example, in our LSI benchmark, one task may characterize diversity using age and hair length, while another uses facial expression and hair color. The same solution can therefore occupy different behavioral niches across tasks. A conventional multi-objective QD method, such as MOME, maintains Pareto-optimal solutions within niches defined by a shared behavior space, but does not directly accommodate these distinct task-specific notions of diversity. This distinction motivates our framework, i.e., MTQD-CoD preserves task-specific QD optimization while enabling complementary exploration and knowledge transfer across tasks. In this work, we consider tasks with potentially different quality objectives and behaviour spaces.

Moreover, the two formulations are not mutually exclusive. Multitask QD can also be extended to multitask multi-objective QD, where multiple multi-objective QD tasks are solved simultaneously and in parallel, each involving multiple quality objectives.

Response to W2 and Q1 (Part 1)

We address these questions by discussing the conditions under which cross-task collaboration is beneficial, followed by additional experiments

(1) When is MTQD-CoD most effective?

As discussed in Section III, a key motivation for MTQD-CoD is to reduce redundant exploration across tasks. Such redundancy can arise when different tasks have overlapping regions of high-quality solutions, or when their search processes tend to repeatedly explore similar regions even if their final solution sets differ. MTQD-CoD uses repulsion to encourage complementary exploration and adaptive transfer to share useful discoveries. Therefore, we expect larger improvements when tasks exhibit large overlapping exploration. If tasks explore largely unrelated regions, there may be little redundant search to avoid, and the benefits of collaboration can be limited.

In fact, this also suggests a natural way to increase the opportunities for collaboration, i.e., jointly optimizing more than two tasks. Our original experiments mainly consider two-task problems. With more tasks, there may be more opportunities to exist overlapping exploration among tasks. We investigate this possibility through an additional four-task LSI experiment below.

(2) Why are the improvements relatively small on Arm?

We agree that performance saturation is an important reason for the relatively small improvements on Arm. First, the Arm problems are relatively easy for the considered QD optimizers, and their performance tends to saturate within the evaluation budget. Second, CMA-MEGA/MAEGA benefits from the strong search capability of CMA-ES in gradient emitter, allowing independent searches to explore diverse solutions efficiently. As a result, the additional benefits of cross-task collaboration can become less visible. To further investigate the effect of cross-task collaborative discovery, we conduct an additional experiment with a smaller initial CMA-ES step size, reducing σ0\sigma_0 from 0.05 to 0.005 for both CMA-MAEGA and MTQD-CoD (MAE). This setting limits the exploration capability of the CMA-ES part.

CMA-MAEGAMTQD-CoD (MAE)
TaskCoverageNorm. QD ScoreCoverageNorm. QD Score
Arm-1 / Task-173.18% ± 1.09%3.66 ± 0.0574.58% ± 0.94%3.73 ± 0.05
Arm-1 / Task-273.47% ± 1.45%3.67 ± 0.0774.52% ± 1.04%3.73 ± 0.05
Arm-2 / Task-137.90% ± 0.58%1.89 ± 0.0338.74% ± 0.63%1.94 ± 0.03
Arm-2 / Task-238.07% ± 0.54%1.90 ± 0.0338.40% ± 0.68%1.92 ± 0.03
Arm-3 / Task-160.39% ± 0.85%3.02 ± 0.0461.31% ± 1.07%3.07 ± 0.05
Arm-3 / Task-261.16% ± 0.83%3.03 ± 0.0461.27% ± 0.97%3.07 ± 0.05

The results are shown in the above table. Under this setting, MTQD-CoD (MAE) achieves a substantially larger improvement over CMA-MAEGA. This observation suggests that cross-task collaboration can provide greater benefits when the independent optimizer has more limited exploration capability. We emphasize that this experiment is intended to investigate when collaboration becomes more valuable, rather than to replace the results under the original configuration.

AuthorsOpenReview ↗

Response — (2/6)

Response to W2 and Q1 (Part 2)

(3) Why are the improvements relatively small on LSI?

For LSI, the relatively small improvements may be attributed to limited task relatedness, as suggested by the observed dynamics of cross-task transfer probabilities shown in Fig. 19. This may indicate less searching overlap (as the limited transfer benefits suggest that the tasks have few shared regions containing high-quality solutions) between task-specific search regions, leaving fewer opportunities for repulsion to reduce redundant exploration. Here, we introduce additional tasks (more than two tasks) to increase the opportunities for overlapping searches across tasks, and examine whether the benefits of collaboration become more pronounced. We extend LSI from two tasks to four jointly QD tasks, including LSI-1/Task 1, LSI-1/Task 2, LSI-2/Task 2, and LSI-3/Task 2 (we do not include LSI-2/Task 1 and LSI-3/Task 1 cause they are the same as the LSI-1 two tasks). We compare MTQD-CoD (MAE) in this four-task setting against independent CMA-MAEGA and the original two-task MTQD-CoD.

TaskCMA-MAEGA / CoverageCMA-MAEGA / Norm. QDMTQD-CoD (MAE, 2 tasks) / CoverageMTQD-CoD (MAE, 2 tasks) / Norm. QDMTQD-CoD (MAE, 4 tasks) / CoverageMTQD-CoD (MAE, 4 tasks) / Norm. QD
LSI-1/Task-151.64% ± 1.65%33.20 ± 1.0453.75% ± 3.18%34.17 ± 1.9554.11% ± 2.05%34.48 ± 1.26
LSI-1/Task-280.03% ± 2.97%52.09 ± 1.8881.86% ± 2.77%52.75 ± 1.9984.14% ± 2.30%54.09 ± 1.50
LSI-2/Task-247.75% ± 2.82%30.12 ± 1.7450.59% ± 1.87%31.62 ± 1.1752.25% ± 1.11%32.73 ± 0.72
LSI-3/Task-278.96% ± 1.93%50.51 ± 1.2780.09% ± 3.33%50.76 ± 2.0983.52% ± 1.61%52.56 ± 0.89

As shown in above table, jointly optimizing four tasks leads to substantially larger improvements than the original two-task setting. We also shows the convergence of the MTQD-CoD (MAE, 4 Tasks) and CMA-MAEGA in Appendix J (page 33) and Fig. 20 (page 32) of the updated manuscript to better show the improvements. These results support our expectation that involving more tasks can create additional opportunities for collaborative exploration and knowledge sharing.

(4) Policy optimization

For policy optimization, the improvements over PPGA are already more pronounced. We therefore focus the above analysis on explaining the relatively small gains observed in Arm and LSI. Regarding the high variance in policy optimization, we provide a separate explanation in our response to Q4.

AuthorsOpenReview ↗

Response — (3/6)

Response to W3:

Thank you for raising these important points. The responses are as follows.

(1) Scalability to More Tasks.

We agree that our original experiments only considered two tasks, leaving the scalability of MTQD-CoD insufficiently explored. Motivated by your suggestion, we conducted additional experiments involving more tasks. As discussed in our response to W2/Q1, MTQD-CoD is expected to benefit from jointly optimizing tasks with overlapping search regions, as this provides more opportunities for complementary exploration and knowledge transfer. Increasing the number of tasks can create additional opportunities for such collaboration. In our response to W2/Q1, we have already presented a four-task LSI experiment, where MTQD-CoD achieves more pronounced improvements than in the original two-task setting. Here, we further extend the Arm experiments to six jointly optimized tasks, including all six tasks from Arm-1, Arm-2, and Arm-3. We compare MTQD-CoD (MAE, 6 Tasks), MTQD-CoD (MAE, 2 Tasks), and independent CMA-MAEGA, under σ0\sigma_0 = 0.005

TaskCMA-MAEGA / CoverageCMA-MAEGA / Norm. QDMTQD-CoD (2 tasks) / CoverageMTQD-CoD (2 tasks) / Norm. QDMTQD-CoD (6 tasks) / CoverageMTQD-CoD (6 tasks) / Norm. QD
Arm-1 / Task-173.18% ± 1.09%3.66 ± 0.0574.58% ± 0.94%3.73 ± 0.0575.41% ± 1.04%3.77 ± 0.05
Arm-1 / Task-273.47% ± 1.45%3.67 ± 0.0774.52% ± 1.04%3.73 ± 0.0575.31% ± 0.95%3.77 ± 0.05
Arm-2 / Task-137.90% ± 0.58%1.89 ± 0.0338.74% ± 0.63%1.94 ± 0.0338.51% ± 0.68%1.93 ± 0.03
Arm-2 / Task-238.07% ± 0.54%1.90 ± 0.0338.40% ± 0.68%1.92 ± 0.0338.58% ± 0.62%1.93 ± 0.03
Arm-3 / Task-160.39% ± 0.85%3.02 ± 0.0461.31% ± 1.07%3.07 ± 0.0561.77% ± 0.95%3.09 ± 0.05
Arm-3 / Task-261.16% ± 0.83%3.06 ± 0.0461.27% ± 0.97%3.06 ± 0.0561.63% ± 0.77%3.08 ± 0.04

The results show that jointly optimizing six tasks can still have the chance to improve the performance.

(2) Task Heterogeneity in Arm-1

We thank the reviewer for pointing out this issue. We agree that, in the original Arm-1 setting, uniformly scaling the link lengths does not change the normalized end-effector coordinates or the quality objective. Consequently, the two tasks are effectively identical optimization problems. To address this issue, we have revised the Arm-1 benchmark by introducing task-specific obstacles, following a similar design to Arm-2 and Arm-3. Specifically, we introduce obstacles occupying approximately 5% of the workspace, with the obstacle located in the upper region for Task-1 and the lower region for Task-2. This modification introduces different constraints into the two tasks, making them genuinely distinct optimization problems rather than equivalent ones. We have rerun the experiments using the revised Arm-1 benchmark and updated the corresponding results in the updated manuscript. (Note that the additional results including the previous provided one are also based on the new Arm-1 settings.) Although the numerical results have changed, the overall conclusions remain consistent with those of the original experiments. In particular, MTQD-CoD continues to demonstrate the benefits of cross-task collaborative discovery, including the additional gains observed when jointly optimizing six tasks.

Finally, we sincerely thank the reviewer for identifying this issue and encouraging us to investigate larger task collections. These suggestions have helped us improve the benchmark design and provide stronger evidence for the effectiveness of MTQD-CoD beyond pairwise collaboration.

Reviewer_fdZDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MTQD-CoD for solving related quality-diversity (QD) tasks jointly. It combines subspace-projected repulsion to encourage complementary exploration with an adaptive inter-task transfer matrix based on archive-improvement feedback. The method is implemented with differentiable QD emitters and evaluated on arm repertoire, latent-space illumination, and MuJoCo policy optimization problems. The reported results generally improve archive coverage or normalized QD scores over the corresponding single-task backbones. However, the contribution can be further enhanced by showing the theoretical benefits of the proposed strategies.

Critical strengths
  1. The paper addresses the meaningful problem of cross-task collaboration in differentiable quality-diversity optimization. The proposed framework combines complementary exploration and adaptive knowledge transfer to reduce redundant search across related tasks. Its modular design also allows integration with existing emitter-based QD algorithms.

  2. The experiments cover three types of QD problems, including arm repertoire, latent-space illumination, and policy optimization. Comparisons with established QD methods demonstrate promising performance in solution quality and diversity. Additional ablation studies and analyses of transfer-probability dynamics further help evaluate the contributions of the proposed mechanisms.

Critical weaknesses
  1. Limited theoretical support and insufficient justification of the proposed mechanisms. The main algorithm combines projected repulsion and adaptive transfer with existing differentiable QD methods, but its methodological novelty relative to earlier evolutionary multitasking and multitask QD approaches is not fully established. The coupon-collector analysis in Appendix A explicitly assumes ideal complementary search, perfect transfer decisions, and uniform sampling; it does not analyze the actual MTQD-CoD updates or establish a convergence or sample-efficiency guarantee. Moreover, repulsion in a local gradient-coefficient space is not necessarily equivalent to reduced overlap in behavioral or archive space. The paper would benefit from more direct evidence linking the proposed mechanisms to the claimed reduction in redundant exploration.
  2. Insufficient evidence for a general and cost-effective cross-task advantage. Although the evaluation includes several domains, all main benchmarks contain only two related tasks, and the comparisons largely rely on single-task QD algorithms rather than competitive joint multitask QD approaches. The ablation results are mixed: removing transfer can outperform the complete method in some arm and latent-space settings (Appendix H), suggesting that the adaptive transfer component is not consistently beneficial. Experiments with more tasks, stronger multitask baselines, transfer-only controls, and wall-clock comparisons are needed to establish generality and practical value.
Questions for authors
  1. Can the authors establish a more rigorous connection between subspace-projected repulsion, adaptive cross-task transfer, and improvements in search efficiency or convergence performance?

  2. Can the authors provide more comprehensive experiments to demonstrate the effectiveness and generality of cross-task collaboration?

9 comments
AuthorsOpenReview ↗

Response — (1/5)

Response to W1 and Q1 (Part 1):

(1) Novelty Compared with EMT and Multitask QD

We would like to highlight two key differences between MTQD-CoD and existing evolutionary multitasking and multitask QD methods.

  1. Existing evolutionary multitasking and multitask QD methods, in most cases, commonly follow an “Attraction + Transfer” strategy. In contrast, our approach takes almost the opposite direction, i.e., “Repulsion + Transfer”. Conventional evolutionary multitasking methods encourage different tasks to search similar promising regions and share useful solutions (we will explain why it is like this below), whereas our method encourages them to explore different regions while still sharing what they discover. This may sound counterintuitive, but it is exactly the central idea behind our cross-task collaborative discovery framework, and also a key methodological contribution that distinguishes it from existing evolutionary multitasking and multitask QD approaches. To further clarify this distinction, we have also added a simple illustrative cartoon (see Fig. 7 on page 18) to the revised manuscript to help readers better understand the difference between the conventional evolutionary multitasking strategy and our proposed “Repulsion + Transfer” strategy.
  2. To the best of our knowledge, existing multitask QD methods have primarily focused on black-box optimization and have not explicitly considered differentiable settings like our work. Existing approaches primarily focus on black-box QD optimization, while our method directly exploits the gradient information available in differentiable QD and introduces cross-task collaboration into the gradient-coefficient search process.

Why “Attraction + Transfer” in conventional evolutionary multitasking, but “Repulsion + Transfer” in our approach?

This is because the two settings emphasize different search goals. Traditional evolutionary multitasking mainly aims to find high-quality solutions faster. When two tasks are related, searching similar promising regions makes sense, since a good solution found by one task may also benefit the other. This is why existing evolutionary multitasking methods often encourage searches toward similar regions through genetic exchange or distribution alignment. Existing multitask QD methods also largely follow this idea. For example, Hou et al. (2025) use VAE-based modeling and alignment to facilitate knowledge sharing across QD tasks. But QD is different. Finding a few high-quality solutions is not enough; we want to discover many diverse, high-quality solutions across the behavior space. If two tasks keep exploring the same promising regions, they may repeatedly discover similar solutions. Even if these solutions are good, much of the exploration becomes redundant. This leads to our main idea, i.e., whether we can let different tasks explore different regions and then share their useful discoveries. This is the motivation behind our “Repulsion + Transfer” strategy, which aims to reduce redundant exploration while retaining the benefits of cross-task knowledge sharing.

(2) Does subspace-projected repulsion reduce redundant exploration?

We thank the reviewer for this important question. We agree that repulsion in the local gradient-coefficient space does not necessarily imply reduced overlap in behavior space. To clarify this relationship, we have added a analysis connecting subspace projected repulsion to behavioral overlap, together with new experiments directly measuring the behavioral search overlap between task-specific emitters. The details can be found in the Appendix L of the updated manuscript.

Additional Analysis. We first define behavioral search overlap (OVL) between two emitters as

\min (p_n^b(y),p_j^b(y)) dy,$$ where $p_n^b$ and $p_j^b$ are their induced behavioral probability densities. Then we consider a local regime in which the two emitters can be analyzed using a common first-order approximation of the behavior function. Taking the target emitter as the reference, the local behavior mapping is $$y(c)=b_n(x_n)+B_nc,\qquad B_n=J_{b_n}(x_n)G_n,$$ where $G_n$ contains both objective and behavior gradients, and $c\sim\mathcal N(\mu_n,\Sigma_n)$ is the gradient-coefficient vector sampled by the target emitter, controlling the linear combination of objective and behavior gradients in $G_n$. Following our subspace projected repulsion mechanism, the source emitter's search distribution is projected into the target coefficient space, yielding $\mathcal N(\mu_{j\to n},\Sigma_{j\to n})$.
AuthorsOpenReview ↗

Response — (2/5)

Response to W1 and Q1 (Part 2):

First, for the local approximated distribution pnb,lin(y(c))p_n^{b,\mathrm{lin}}(y(c)) and pj→nb,lin(y(c))p_{j\to n}^{b,\mathrm{lin}}(y(c)) under the local linear behavior mapping, we show that behavioral overlap can be expressed as an expectation over target offspring:

\left[ \min ( 1, \frac{ p_{j\to n}^{b,\mathrm{lin}}(y(c)) }{ p_n^{b,\mathrm{lin}}(y(c)) } ) \right].$$ This suggests that, in evolutionary search, reducing behavioral overlap can be encouraged by preferentially selecting target offspring located in regions with lower source behavior density. Under the local linear-Gaussian model, we further obtain $$p_{j\to n}^{b,\mathrm{lin}}(y(c)) \propto \exp\left(-\frac12s_b^{(j)}(c)\right),$$ where the behavioral repulsion score based on the OVL under local linear apporximation is $$s_b^{(j)}(c) = (c-\mu_{j\to n})^\top B_n^\top (B_n\Sigma_{j\to n}B_n^\top)^{-1} B_n(c-\mu_{j\to n}).$$ Thus, a larger behavioral repulsion score corresponds to lower source behavior density. Second, we establish a relationship between this behavioral repulsion and our proposed repulsion score. Recall that repulsion score $$s_{rep}^{(j)}(c)=(c-\mu_{j\to n})^\top \Sigma_{j\to n}^{-1} (c-\mu_{j\to n}).$$ By decomposing the whitened coefficient difference into behavior-sensitive and behavior-insensitive components, we obtain $$s_{rep}^{(j)}(c)=s_b^{(j)}(c) + |(I-\Pi)z|^2 \geq s_b^{(j)}(c),$$ where $z=\Sigma_{j\to n}^{-1/2}(c-\mu_{j\to n})$, and $\Pi$ is the orthogonal projector onto the behavior-sensitive subspace. The above result shows that the proposed respulsion explicitly contains the behavioral repulsion component. When offspring separation is primarily behavior-sensitive, subspace projected repulsion closely approximates behavioral repulsion, favoring offspring with lower source behavior density and thereby encouraging complementary exploration. Therefore, subspace projected repulsion serves as a reseaonable surrogate for reducing redundant exploration. **Experimental Validation.** To examine whether the designed mechanism reduces behavioral overlap in practice, we introduce an empirical OVL metric. At each iteration, we record the 36 offspring generated by each emitter before cross-task transfer, excluding transferred solutions and DQD-stage solutions. We aggregate offspring over the latest five generations and map them into the normalized end-effector behavior space, discretized into a $50\times50$ grid. Given the resulting empirical distributions $p$ and $q$, we compute $$\sum_{m=1}^{2500}\min(p_m,q_m).$$ A smaller OVL indicates less redundant behavioral exploration. We average OVL and report the mean and standard deviation across 20 independent runs as follows. | Algorithm | Arm-1 OVL (×10⁻²) ($\downarrow$) | Arm-3 OVL (×10⁻²) ($\downarrow$) | | -------------- | --------------------------------:| --------------------------------:| | CMA-MEGA | 8.20 ± 0.61 | 7.77 ± 0.66 | | MTQD-CoD (ME) | **6.61 ± 0.91** | **6.93 ± 0.80** | | CMA-MAEGA | 6.40 ± 0.45 | 6.19 ± 0.50 | | MTQD-CoD (MAE) | **5.61 ± 0.56** | **5.43 ± 0.69** | The results show that MTQD-CoD consistently achieves lower behavioral overlap than its corresponding independent QD baselines. Specifically, MTQD-CoD (ME) reduces OVL by 19.4% on Arm-1 and 10.8% on Arm-3, while MTQD-CoD (MAE) achieves reductions of 12.3% on both problems. These findings provide direct empirical evidence that MTQD-CoD encourages different emitters to explore more complementary behavioral regions. Together, the new analysis establishes how subspace projected repulsion incorporates behavioral repulsion, while the empirical results demonstrate reduced redundant exploration in the actual evolutionary search process. We have incorporated both analyses into the revised manuscript. We agree that a rigorous convergence or sample-efficiency guarantee for the complete MTQD-CoD framework is challenging, particularly due to the interactions between CMA-based emitter updates, subspace projected repulsion-guided selection, and adaptive cross-task transfer. We have explicitly acknowledge this limitation in the revised manuscript (Appendix M).
AuthorsOpenReview ↗

Response — (3/5)

Response to W2 and Q2 (Part 1)

To further evaluate the generality and practical value of MTQD-CoD, we conducted additional experiments involving more tasks, a competitive multitask QD baseline, fixed-probability transfer, and computational overhead measurements. We present the new experimental results below and discuss the mixed ablation results.

(1) Scalability to More Tasks

We agree that the original experiments mainly considered two-task settings. To investigate whether MTQD-CoD can benefit from collaboration among more tasks, we conducted two new experiments involving four jointly optimized LSI tasks and six jointly optimized Arm tasks.

Four-task LSI. To evaluate MTQD-CoD beyond the original two-task settings, we conducted an additional experiment jointly optimizing four LSI tasks: LSI-1/Task 1, LSI-1/Task 2, LSI-2/Task 2, and LSI-3/Task 2. We compared MTQD-CoD (MAE, 4 Tasks) against independent CMA-MAEGA and the original two-task MTQD-CoD under the same per-task evaluation budget.

TaskCMA-MAEGA / CoverageCMA-MAEGA / Norm. QDMTQD-CoD (MAE, 2 tasks) / CoverageMTQD-CoD (MAE, 2 tasks) / Norm. QDMTQD-CoD (MAE, 4 tasks) / CoverageMTQD-CoD (MAE, 4 tasks) / Norm. QD
LSI-1/Task-151.64% ± 1.65%33.20 ± 1.0453.75% ± 3.18%34.17 ± 1.9554.11% ± 2.05%34.48 ± 1.26
LSI-1/Task-280.03% ± 2.97%52.09 ± 1.8881.86% ± 2.77%52.75 ± 1.9984.14% ± 2.30%54.09 ± 1.50
LSI-2/Task-247.75% ± 2.82%30.12 ± 1.7450.59% ± 1.87%31.62 ± 1.1752.25% ± 1.11%32.73 ± 0.72
LSI-3/Task-278.96% ± 1.93%50.51 ± 1.2780.09% ± 3.33%50.76 ± 2.0983.52% ± 1.61%52.56 ± 0.89

The results show that four-task MTQD-CoD achieves higher Coverage and Normalized QD Score than both baselines on all four tasks. For example, on LSI-3/Task 2, Coverage increases from 78.96% with CMA-MAEGA to 83.52%, while Normalized QD Score improves from 50.51 to 52.56. Jointly optimizing four tasks consistently outperforms the original two-task MTQD-CoD, suggesting that involving more tasks can further improve collaborative discovery. The reasons are explained as follows. As discussed in Section III, the main benefit of MTQD-CoD comes from reducing redundant exploration when tasks have overlapping search regions or high-quality solution sets. In the original two-task LSI settings, the tasks can differ considerably, leaving limited opportunities for such overlap. By jointly optimizing more tasks, we increase the opportunities for overlapping exploration and useful knowledge sharing, which may explain the stronger improvements observed in the four-task setting. We have also added the corresponding convergence curves to the revised manuscript.

Comments on the paper

2 comments

Dear Reviewers, the authors have submitted their response. Kindly review it and update your review where appropriate.

AuthorsOpenReview ↗

Summary of Major Reviewer Concerns and Our Responses

Dear Area Chair and Reviewers,

We thank the reviewers for their constructive feedback and discussion. We believe the rebuttal addresses most of the concerns raised. In response to pL1p and FceS, we added comparisons with independent CMA-MAEGA at the same evaluation budget, per-seed PPGA results, and wall-clock measurements. To address pL1p’s and fdZD’s questions about collaboration beyond two tasks, we added four-task LSI and six-task Arm experiments and corrected Arm-1 so that its two tasks have distinct constraints. We also investigated behavioral overlap and compared adaptive transfer with fixed-probability transfer in response to questions from fdZD and FceS. Reviewer pL1p indicated that our responses addressed most of their concerns and subsequently raised their score.

We appreciate fdZD’s request for stronger theoretical justification. We acknowledge that end-to-end convergence remains open for MTQD-CoD and is an important direction for future work. Foundational work in differentiable QD illustrates the challenge. CMA-MEGA introduced gradient-based emitters and evaluated their performance empirically. Subsequent work on CMA-MAEGA established specific properties of archive annealing under assumptions such as a fixed candidate sequence or a constant objective (Zhao et al. [r1], Section 5, Theorems 5.3–5.4). Extending these insights to end-to-end convergence is a substantial theoretical step. In MTQD-CoD, the analysis must also account for how emitter adaptation, archive updates, and adaptive transfer interact across tasks. The local analysis and additional experiments provide evidence for the proposed coordination mechanism and its practical value in the settings studied.

In terms of the novelty, our method MTQD-CoD makes two contributions. First, it introduces cross-task collaborative discovery. Prior approaches such as HMQD enable knowledge transfer across task-specific archives. MTQD-CoD also explicitly coordinates where tasks search, encouraging complementary exploration while adaptively sharing useful discoveries. Second, to our knowledge, MTQD-CoD maybe the first framework specifically designed for multitask differentiable QD. The same design can be integrated into CMA-MEGA, CMA-MAEGA, and PPGA, with particularly clear gains over independent PPGA in policy optimization.

The main additional additions during the rebuttal were:

  • Repulsion and adaptive transfer. Appendix L characterizes when subspace-projected repulsion can encourage separation in behavior space under a local approximation. On Arm-1 and Arm-3, MTQD-CoD reduced behavioral overlap by 10.8%–19.4% relative to the corresponding independent baselines, measured on offspring generated before transfer. On the conflicting Arm-2 tasks, the learned transfer probabilities decreased, and adaptive transfer improved coverage and normalized QD score over the tested fixed-transfer setting.

  • Performance and computational cost. Paired Wilcoxon tests supported the improvements over independent CMA-MAEGA on all four tasks in the new LSI experiment and on five of the six tasks in the additional Arm setting. The two collaboration mechanisms accounted for 0.82% of the measured runtime on the tested LSI benchmark. On Humanoid, independent PPGA fell below 18% coverage in two task–seed runs, while all ten MTQD-CoD runs exceeded 55%. MTQD-CoD retained a higher mean normalized QD score on both tasks after excluding the two corresponding pairs of runs.

  • Broader applicability. The four-task LSI and six-task Arm experiments extend the evaluation beyond pairwise collaboration. In the separate exploratory SQUAD study discussed with pL1p, adapting the same collaboration principle improved QD scores by approximately 19.5% on both homogeneous tasks with 16-dimensional behavior spaces at the same total evaluation budget.

Thank you for considering the revised submission and the additional evidence from the discussion.

Best regards,
The authors

References

[r1] Zhao, Shihan, et al. "Covariance matrix adaptation MAP-annealing: Theory and experiments." ACM Transactions on Evolutionary Learning 5.1 (2025): 1-53.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.