- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Comparing complete meta-black-box optimizers makes it difficult to identify which design choices explain their performance. EvoX MetaBench exposes interchangeable components within shared evaluation workflows, supported by batched GPU execution. Its controlled configuration studies illustrate how this framework can separate the benefits of configuration diversity from those of learned control.
Critical strengths
-
The framework makes controlled component substitution practical. Appendix B gives task-specific interfaces and assembly examples that allow researchers to change a component while preserving the surrounding procedure. This provides a useful foundation for attributing performance differences to optimizer design choices.
-
RQ3 usefully separates configuration diversity from learned control. Its random population-level and individual-level variants show how heterogeneous configurations can improve search before learning is introduced. This demonstrates the kind of insight that the component-based framework is designed to support.
Critical weaknesses
-
My main concern is whether the reimplemented baselines behave like the original methods. Appendices B–C describe their assembly, but I could not find validation against the original implementations. Without that check, it is hard to interpret the rankings in Tables 5–6 as comparisons of the named methods. This limits confidence in the benchmark results, although the component interfaces remain a useful contribution.
-
It is unclear how stable the learned-component findings are across independent training runs. The authors appropriately limit the population-size observations to the fixed training budget. Even within that condition, the 30 optimization test runs in Appendices F–G characterize the evaluated checkpoints; the number of independent training runs is unspecified. The observations are useful, but their reliability as guidance about configuration granularity and state representation remains uncertain. This limits the generality of the empirical insights more than the value of the framework itself.
Questions for authors
-
What evidence shows that the assembled baseline recipes preserve the behavior of the original methods under matched evaluation conditions? A representative comparison would help establish how to interpret the reported rankings.
-
How many independent meta-training runs underlie RQ3 and RQ4, and how consistent are the main findings across those runs under the stated fixed training budget? This would clarify the strength of the resulting design guidance.
8 comments
W2/Q2 (Consistency across independent meta-training runs).
How many independent training runs? The original RQ3 and RQ4 results used one meta-training run (seed 42) per learned setting. To further assess the consistency of these findings across independent meta-training runs, we have added two training seeds (43 and 44), giving three training runs per setting under the same budget of 100 episodes on eight training functions. Each checkpoint is evaluated on 16 held-out 40D BBOB functions with 30 test seeds per function. Appendices G.4 and H.4 report the results separately for each training seed.
RQ3: How consistent is the effect of configuration granularity? We retrained both Meta-Pop and Meta-Ind at all four population sizes. The table counts the functions on which Meta-Ind achieves lower mean final best-so-far fitness than Meta-Pop.
| Population | Seed 42 | Seed 43 | Seed 44 |
|---|---|---|---|
| 20 | 14/16 | 11/16 | 15/16 |
| 50 | 14/16 | 11/16 | 14/16 |
| 100 | 6/16 | 7/16 | 7/16 |
| 200 | 9/16 | 8/16 | 8/16 |
Meta-Ind retains its advantage on most functions at and across all three training seeds. At , Meta-Pop instead performs better on 9–10 functions; at , the comparison is nearly even. These results support the population-size dependence reported in RQ3: individual-level control offers a consistent advantage at smaller populations, but this advantage does not extend uniformly to larger populations under the fixed training budget.
RQ4: How consistent are the state-representation findings? We independently retrained DDQN for all eight representations with DE, PSO, and ES at . The pretrained NeurELA encoder remains frozen. For each optimizer and training seed, the table shows the representation with the most function-level wins among the eight variants. A win denotes the lowest mean final best-so-far fitness; counts are out of 16 functions. F and P denote fitness-space and optimization-process information, respectively.
| Optimizer | Seed 42 | Seed 43 | Seed 44 |
|---|---|---|---|
| DE | F (13/16) | F+P (15/16) | P (13/16) |
| PSO | NeurELA (8/16) | F+P (8/16) | F+P (10/16) |
| ES | F (7/16) | F+P (9/16) | F (12/16) |
The leading representation changes across training seeds for all three optimizers. For PSO, F+P leads under both additional seeds, while NeurELA leads under the original seed. These results support the observation that representation effectiveness depends on the optimizer and further show sensitivity to training randomness. More broadly, a state design that performs well in one training run may not retain its advantage after retraining. Claims about its effectiveness should therefore be supported by comparisons across independent training runs. We thank the reviewer for this insightful suggestion, which prompted this analysis and helped clarify an evaluation issue of broader relevance to MetaBBO research.
Follow-up Questions on W1
Thank you for the additional validation. However, reporting only the median of the absolute standardized differences removes their direction and hides variation across functions. Please provide the full per-function results, including signed differences, to show whether EvoX systematically outperforms or underperforms the original implementation and whether individual functions show substantial discrepancies.
Values such as 0.157 are also difficult to interpret without a reference for variation within the original implementation itself. I suggest comparing two independent batches of 30 runs from the original implementation using the same metric, repeating this to establish a reference distribution. The Original–EvoX comparison should follow the same sampling protocol. This would help assess whether cross-implementation differences exceed the variation typically observed when repeating experiments with the original implementation.
Thank you for the careful review and suggestions on baseline validation and variability across training runs. We address the two points below.
What did we compare? We compared EvoX MetaBench with the original authors' implementations and MetaBox-v2 for two representative methods: LDE (RL-based algorithm configuration) and GLHF (SL-based solution manipulation). For each method, we transferred the same checkpoint trained in EvoX MetaBench to the other two implementations and kept its parameters fixed. All three implementations used identical initial populations on the same 16 held-out 40D BBOB functions, with 30 paired test seeds per function and 10,000 function evaluations per run.
How closely do the results agree? For each function, we divide the absolute difference in mean final best-so-far fitness by the pooled standard deviation of the two implementations. The table reports the median across the 16 functions; lower values indicate closer agreement.
For both methods, the median Original–EvoX difference is small relative to variation across optimization runs. These results provide evidence of comparable final optimization performance between our assemblies and the original implementations under the matched test conditions.
What does this validation show? Evaluating the same frozen checkpoint across implementations isolates the effect of implementation differences from variability introduced by retraining. The results provide representative evidence that the assembled LDE and GLHF recipes retain comparable final optimization performance to the original implementations under matched evaluation conditions.
How are the baseline recipes constructed? We prioritize the original authors' released implementations as the reference for modularization. When these are unavailable, we follow the original papers and consult third-party implementations from established benchmarks such as MetaBox and PlatMetaX. If no public implementation is available, we reconstruct the method from its published algorithm description. We then express the method through task-specific components and assemble it into the shared training and evaluation workflow.