- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a multitask differentiable quality-diversity optimization framework that encourages complementary exploration and adaptive knowledge transfer across related tasks. The motivation is intuitive, and the method is relatively modular, combining subspace-projected repulsion with adaptive inter-task transfer probabilities. Experiments on arm repertoire, latent-space illumination, and policy optimization tasks demonstrate improvements over the corresponding single-task baselines.
Critical strengths
The motivation is intuitive and easy to follow. Reducing redundant exploration while sharing useful discoveries across related tasks is a reasonable objective. The proposed framework also provides a relatively straightforward way to incorporate these ideas into existing differentiable QD algorithms.
Critical weaknesses
-
Limited clarity on novelty. While I am not very familiar with the multitask QD literature, the central idea of reducing redundant exploration and transferring useful solutions across related tasks seems relatively natural in multitask optimization. I am not fully convinced that the paper sufficiently distinguishes its contributions from existing multitask QD or evolutionary multitasking approaches.
-
Limited performance gains over strong baselines. On the Arm Repertoire and LSI benchmarks, the improvements over CMA-MAEGA are generally modest. Although the gains are more substantial in policy optimization, MTQD-CoD still underperforms DCG-ME on Ant. It is unclear whether the improvements justify the additional computational overhead.
-
Insufficient evidence for the proposed mechanisms. It is not entirely clear whether the proposed repulsion mechanism actually reduces redundant exploration, or whether adaptive transfer provides substantial benefits over simpler fixed-probability transfer strategies. The effectiveness of these mechanisms when tasks are weakly related or conflicting also remains unclear. These issues are important for understanding whether both components are necessary.
Questions for authors
-
Could the authors clarify the novelty of MTQD-CoD relative to existing multitask QD and evolutionary multitasking methods?
-
What is the additional computational overhead of MTQD-CoD, and how does it compare with the observed performance gains?
-
Can the authors provide more direct evidence that repulsion reduces search overlap and that adaptive transfer outperforms fixed transfer probabilities, particularly for weakly related tasks?
9 comments
Response — (2/4)
Response to W2 and Q2 (Part 1)
Thank you for your comments. To further demonstrate the effectiveness of MTQD-CoD and clarify when cross-task collaboration provides greater benefits, we conducted additional experiments on Arm and LSI. The new results show more improvements over CMA-MAEGA, particularly in the four-task LSI setting, where MTQD-CoD consistently achieves better Coverage and Normalized QD Scores across all tasks. We also provide a computational complexity introduction and wall-clock measurements to evaluate the additional overhead of the proposed mechanisms.
(1) Performance improvements on Arm
The relatively small improvements on Arm in the previous manuscript are partly due to performance saturation. These problems are relatively easy for CMA-MAEGA, whose CMA-ES-based gradient emitter already provides strong exploration capabilities, leaving limited room for additional improvement. To further investigate this, we conducted a new experiment by reducing the initial CMA-ES step size from to for both CMA-MAEGA and MTQD-CoD (MAE). This allows us to examine whether cross-task collaboration becomes more beneficial when the exploration capability of the independent optimizer is more constrained.
| CMA-MAEGA | MTQD-CoD (MAE) | |||
|---|---|---|---|---|
| Task | Coverage | Norm. QD Score | Coverage | Norm. QD Score |
| Arm-1 / Task-1 | 73.18% ± 1.09% | 3.66 ± 0.05 | 74.58% ± 0.94% | 3.73 ± 0.05 |
| Arm-1 / Task-2 | 73.47% ± 1.45% | 3.67 ± 0.07 | 74.52% ± 1.04% | 3.73 ± 0.05 |
| Arm-2 / Task-1 | 37.90% ± 0.58% | 1.89 ± 0.03 | 38.74% ± 0.63% | 1.94 ± 0.03 |
| Arm-2 / Task-2 | 38.07% ± 0.54% | 1.90 ± 0.03 | 38.40% ± 0.68% | 1.92 ± 0.03 |
| Arm-3 / Task-1 | 60.39% ± 0.85% | 3.02 ± 0.04 | 61.31% ± 1.07% | 3.07 ± 0.05 |
| Arm-3 / Task-2 | 61.16% ± 0.83% | 3.03 ± 0.04 | 61.27% ± 0.97% | 3.07 ± 0.05 |
The new results show that MTQD-CoD outperforms CMA-MAEGA in both Coverage and Normalized QD Score on all six Arm tasks, with larger improvements than those observed under the original configuration. This suggests that cross-task collaboration provides greater benefits when independent exploration is more limited.
(2) Performance improvements on LSI.
As discussed in Section III, the main benefits of MTQD-CoD come from reducing redundant exploration when different tasks have overlapping search regions or high-quality solution sets. However, the two tasks in some LSI settings differ considerably in their objectives and behavior descriptors, making such overlap less likely. This is also consistent with the observed dynamics of cross-task transfer probabilities in Fig. 19 which suggest relatively limited task relatedness in these settings. Consequently, there may be fewer opportunities for complementary exploration and useful knowledge transfer, which may explain the relatively modest improvements observed in the original experiments. A natural way to increase the opportunities for collaboration is to jointly optimize more tasks, since a larger task set may increase the chance that some tasks share overlapping search regions or useful discoveries. Motivated by this, we conducted an additional experiment that jointly optimizes four LSI tasks to investigate whether broader cross-task collaboration can lead to more substantial improvements. We therefore extended the experiments to jointly optimize four LSI tasks and compared MTQD-CoD (MAE, 4 Tasks) against independent CMA-MAEGA and the original two-task MTQD-CoD.
Response — (3/4)
Response to W2 and Q2 (Part 2)
| Task | CMA-MAEGA / Coverage | CMA-MAEGA / Norm. QD | MTQD-CoD (MAE, 2 tasks) / Coverage | MTQD-CoD (MAE, 2 tasks) / Norm. QD | MTQD-CoD (MAE, 4 tasks) / Coverage | MTQD-CoD (MAE, 4 tasks) / Norm. QD |
|---|---|---|---|---|---|---|
| LSI-1/Task-1 | 51.64% ± 1.65% | 33.20 ± 1.04 | 53.75% ± 3.18% | 34.17 ± 1.95 | 54.11% ± 2.05% | 34.48 ± 1.26 |
| LSI-1/Task-2 | 80.03% ± 2.97% | 52.09 ± 1.88 | 81.86% ± 2.77% | 52.75 ± 1.99 | 84.14% ± 2.30% | 54.09 ± 1.50 |
| LSI-2/Task-2 | 47.75% ± 2.82% | 30.12 ± 1.74 | 50.59% ± 1.87% | 31.62 ± 1.17 | 52.25% ± 1.11% | 32.73 ± 0.72 |
| LSI-3/Task-2 | 78.96% ± 1.93% | 50.51 ± 1.27 | 80.09% ± 3.33% | 50.76 ± 2.09 | 83.52% ± 1.61% | 52.56 ± 0.89 |
As shown in the new results, the four-task MTQD-CoD achieves higher Coverage and Normalized QD Score than both baselines on all four tasks. For example, on LSI-3/Task 2, Coverage increases from 78.96% with CMA-MAEGA to 83.52%, while Normalized QD Score increases from 50.51 to 52.56. The improvements are also more obvious than in the original two-task setting. We have further added the corresponding convergence curves in Appendix J, Figure 20 (page 32) to show how these improvements develop throughout optimization. These results show higher performance gains than the strong baselines.
(3) Performance on Ant.
Regarding policy optimization, MTQD-CoD is developed by extending PPGA to the multitask setting and therefore inevitably inherits some of the limitations of its underlying QD optimizer. Ant presents a more challenging exploration setting, with a four-dimensional behavior space defined by the contact frequencies of its four legs. More importantly, Batra et al. (2024, Section 5, p. 9) explicitly acknowledge that PPGA's adaptive action standard deviation can impair exploration on Ant as PPO may reduce policy variance during training, limiting the discovery of new archive cells. This inherited limitation may partly explain why both PPGA and MTQD-CoD perform less competitively than DCG-ME on Ant. Nevertheless, MTQD-CoD consistently improves upon its PPGA backbone in mean Normalized QD Score on both Ant tasks demonstrating that cross-task collaborative discovery can provide additional benefits even when the underlying QD optimizer has limitations on a particular benchmark.
[r2] Batra, Sumeet, et al. "Proximal policy gradient arborescence for quality diversity reinforcement learning." International Conference on Learning Representations. Vol. 2024. 2024.
(4) Computational overhead
To evaluate the additional computational cost of MTQD-CoD, we provide both a complexity analysis and new wall-clock measurements. In terms of subspace-projected repulsion, although the decision space can be high-dimensional (e.g., 7,168 dimensions in LSI), subspace-projected repulsion operates in a much smaller gradient-coefficient space of dimension , where is the behavior descriptor dimension. In our experiments, this dimension is only 3–5. The additional operations mainly involve low-dimensional covariance projections and Mahalanobis distance calculations, with a computational complexity of
where , , and denote the number of tasks, decision-space dimensionality, and batch size, respectively. In terms of adaptive transfer, it only requires categorical sampling and updates to an transfer probability matrix. Moreover, candidates are reassigned across tasks rather than generating additional candidates, so the total evaluation budget remains unchanged. We further measured the running time of both mechanisms on Arm and LSI. The results are shown below.
| Benchmark | Iterations | Total runtime (s) | Subspace Projected Repulsion (s) | Adaptive Transfer (s) |
|---|---|---|---|---|
| Arm (A1) | 2000 | 9.98 | 0.819 | 0.669 |
| LSI (P3) | 1000 | 306.12 | 1.962 | 0.563 |
These measurements allow us to assess the additional computational cost introduced by cross-task repulsion and adaptive transfer alongside the observed performance improvements.
Response — (1/4)
Response to W1 and Q1
We thank the reviewer for the constructive comments. We would like to clarify that explicitly reducing redundant exploration across tasks is not common in conventional multitask optimization or multitask QD. In fact, our approach is somewhat counterintuitive compared with conventional multitask optimization, as conventional multitask optimization methods (in most cases) commonly encourage different tasks to explore similar regions rather than different ones (we will explain why this happened in below). We believe this represents a key conceptual novelty of MTQD-CoD, which introduces a “Repulsion + Transfer” strategy instead of the more conventional multitask optimization's “Attraction + Transfer” paradigm. To further clarify this distinction, we have also added a simple illustrative cartoon to the revised manuscript to help readers better understand the difference between the conventional multitask optimization strategy and our proposed “Repulsion + Transfer” strategy (see Fig. 7 on page 18). Furthermore, to the best of our knowledge, MTQD-CoD maybe the first attemption specifically designed for differentiable settings (in multitask QD), an area that remains unexplored. Existing multitask QD methods primarily focus on black-box optimization, whereas our approach explicitly exploits gradient information to enable complementary exploration and adaptive knowledge transfer.
Why is “Repulsion + Transfer” different from conventional multitask optimization?
The key difference lies in the search purpose. Conventional multitask optimization primarily aims to find high-quality solutions more efficiently. When tasks objective landscape are (positive) correlated, encouraging them to search similar promising regions can facilitate the improvement on the convergence, since promising regions searched by one task may also guide another one to identify high-quality solutions. This motivates common multitask optimization mechanisms such as genetic exchange and distribution alignment. Existing multitask QD methods also largely emphasize knowledge sharing and alignment. For example, Hou et al. (2025) [r1] employ VAE-based modeling and alignment to facilitate cross-task transfer. However, QD optimization aims to discover many diverse, high-quality solutions rather than only a few high-quality ones. If multiple tasks repeatedly explore the same promising regions, they may rediscover similar solutions, resulting in redundant search efforts. This motivates our alternative strategy, i.e., encouraging different tasks to explore complementary regions while still sharing their useful discoveries
[r1] Hou, Yaqing, et al. "Evolutionary heterogeneous multitasking for quality diversity optimization." IEEE Transactions on Evolutionary Computation (2025).