- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
HATS selects training tasks for meta black-box optimization using the performance gap between the learner and a baseline optimizer. A bandit selects function classes, while prioritized replay selects instances. Experiments cover multiple optimization backbones and synthetic and real-world tasks.
Critical strengths
- The method addresses a useful gap between task diversity and training value, offering a practical curriculum for learned optimizers.
- The method improves results across multiple optimization backbones and task types.
Critical weaknesses
- The utility proxy is not directly validated against training value. The diversity-quality study does not show that teacher-relative gaps predict learning progress, leaving the central selection principle insufficiently supported.
- The efficiency evidence is limited to training episodes. It is unclear how task-screening trajectories and reference-optimizer evaluations are counted, so the benefit under a matched total evaluation budget remains uncertain.
Questions for authors
See the weaknesses above.
3 comments
Thanks for the detailed responses and additional experiments. The new results address my concern. I am keeping my weak acceptance recommendation.
Dear Reviewer rLAY,
Thanks for your feedback! We are glad to hear that your concerns have been addressed. We sincerely appreciate the time and effort you have dedicated to reviewing our paper and providing thoughtful and valuable comments.
Best regards,
Authors of Submission 712
Thanks for your encouraging comments. Below please find our responses.
W1 Validation of the utility proxy
Thanks for your helpful comments. We agree that the diversity–quality study alone does not show that teacher-relative gaps predict learning progress. To validate whether this proxy translates into actual learning gains, we conduct two illustrative experiments accordingly.
Firstly, we visualize the relationship between different scoring metrics and LOO quality in https://anonymous.4open.science/r/hats-rebuttal-figures-8E48/loo_validation_3panel.pdf. We start from a fixed mid-training checkpoint, randomly generate 32 instances, and compare three metrics: 1) our proposed teacher-student gap utility in Eq. (4), 2) the landscape latent diversity in Eq. 2, and 3) random score. Here we can observe that our proposed utility exhibits a stronger correlation with LOO quality (Pearson r=0.734, Spearman ρ=0.675) than latent diversity (r=0.184, ρ=0.222). Its Spearman correlation also exceeds the central 95% range of correlations obtained from 200,000 random scores ([−0.350,0.349]), whereas the diversity correlation falls within this range. This provides clear evidence that our proposed teacher-student performance gap utility offers a better predictive signal towards the instances' quality than the diversity-based scoring.
Additionally, we directly examine whether the utility predicts subsequent learning gains through controlled continuation experiments. Specifically, we resume from checkpoints at epochs 25, 50, and 75 and group tasks according to their teacher–student log-gap utility U : too-easy (U<−0.5), in-zone ( −0.5≤U≤4.0 ), and too-hard ( U>4.0 ). For each checkpoint, we randomly select 10 tasks from each zone and continually train each model for 1,000 episodes. The following table shows the performance gains over the base policy on held-out test sets, where the in-zone tasks yield the largest gains at all three checkpoints.
These results provide supportive evidence that the teacher-relative gaps offer predictive signals towards better instance quality. We have added these experiments to Section 5.2 (RQ5, Figure 5) and Appendix D.1 (Table 3). Thank you very much.
W2 Evaluation under matched budgets
Thanks for your good question. It is worth comparing different methods under the same training-time fitness evaluation budgets, which include both learner and teacher-baseline evaluations. The additional evaluation budgets for different MetaBBO backbones are different:
We have added these experiments and checkpoint-alignment details to Section 5.2 (RQ2) and Appendix D.7 (Figure 11). Thank you very much.
[1] Symbol: Generating Flexible Black-Box Optimizers through Symbolic Equation Learning. ICLR 2024.
[2] Learning adaptive differential evolution algorithm from optimization experiences by policy gradient. IEEE TEvC 2021.
[3] B2opt: Learning to optimize black-box optimization with little budget. AAAI 2025.
We hope our response has addressed your concerns. If we have missed anything, please let us know.