acceptodds
Under review as a conference paper at ICLR 2027

EvoX MetaBench: Composable Benchmarking for Meta-Black-Box Optimization at Scale

Abstract

Meta-black-box optimization (MetaBBO) automates optimizer design through learning or design search. However, method-specific implementations often couple components, which complicates controlled comparisons of their effects and interactions. Such comparisons also incur substantial computational costs from repeated optimization runs across problems and random seeds. To address these challenges, we introduce EvoX MetaBench, a composable framework and benchmark for MetaBBO at scale. Its task-specific interfaces support component assembly for algorithm selection, algorithm configuration, solution manipulation, and algorithm generation. Researchers can build custom workflows or instantiate baseline recipes assembled from the framework's components, then substitute and recombine compatible components under shared experimental conditions. The benchmark provides over 11,000 problem instances and instance-generation recipes across numerical, combinatorial, and real-world optimization, with explicit training, validation, and test manifests for reproducible comparisons. To support these workflows at scale, the framework builds on EvoX to batch optimization runs on GPUs. Experiments show that GPU execution accelerates LDE training and testing by up to and , respectively, compared with CPU execution. Component studies show that configuration gains depend on population size, useful state information depends on the optimizer, and the link between parent and offspring quality depends on the generation operator.

Primary area:
datasets and benchmarks
Keywords:
Meta-Black-Box Optimization, Benchmarking Platform, GPU-acceleration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_hDzjOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Comparing complete meta-black-box optimizers makes it difficult to identify which design choices explain their performance. EvoX MetaBench exposes interchangeable components within shared evaluation workflows, supported by batched GPU execution. Its controlled configuration studies illustrate how this framework can separate the benefits of configuration diversity from those of learned control.

Critical strengths
  1. The framework makes controlled component substitution practical. Appendix B gives task-specific interfaces and assembly examples that allow researchers to change a component while preserving the surrounding procedure. This provides a useful foundation for attributing performance differences to optimizer design choices.

  2. RQ3 usefully separates configuration diversity from learned control. Its random population-level and individual-level variants show how heterogeneous configurations can improve search before learning is introduced. This demonstrates the kind of insight that the component-based framework is designed to support.

Critical weaknesses
  1. My main concern is whether the reimplemented baselines behave like the original methods. Appendices B–C describe their assembly, but I could not find validation against the original implementations. Without that check, it is hard to interpret the rankings in Tables 5–6 as comparisons of the named methods. This limits confidence in the benchmark results, although the component interfaces remain a useful contribution.

  2. It is unclear how stable the learned-component findings are across independent training runs. The authors appropriately limit the population-size observations to the fixed training budget. Even within that condition, the 30 optimization test runs in Appendices F–G characterize the evaluated checkpoints; the number of independent training runs is unspecified. The observations are useful, but their reliability as guidance about configuration granularity and state representation remains uncertain. This limits the generality of the empirical insights more than the value of the framework itself.

Questions for authors
  1. What evidence shows that the assembled baseline recipes preserve the behavior of the original methods under matched evaluation conditions? A representative comparison would help establish how to interpret the reported rankings.

  2. How many independent meta-training runs underlie RQ3 and RQ4, and how consistent are the main findings across those runs under the stated fixed training budget? This would clarify the strength of the resulting design guidance.

8 comments

Thank you for the careful review and suggestions on baseline validation and variability across training runs. We address the two points below.

W1/Q1 (Validation of the assembled baseline implementations).

What did we compare? We compared EvoX MetaBench with the original authors' implementations and MetaBox-v2 for two representative methods: LDE (RL-based algorithm configuration) and GLHF (SL-based solution manipulation). For each method, we transferred the same checkpoint trained in EvoX MetaBench to the other two implementations and kept its parameters fixed. All three implementations used identical initial populations on the same 16 held-out 40D BBOB functions, with 30 paired test seeds per function and 10,000 function evaluations per run.

How closely do the results agree? For each function, we divide the absolute difference in mean final best-so-far fitness by the pooled standard deviation of the two implementations. The table reports the median across the 16 functions; lower values indicate closer agreement.

MethodOriginal–EvoXOriginal–MetaBoxMetaBox–EvoX
LDE0.1570.1530.156
GLHF0.2670.4880.535

For both methods, the median Original–EvoX difference is small relative to variation across optimization runs. These results provide evidence of comparable final optimization performance between our assemblies and the original implementations under the matched test conditions.

What does this validation show? Evaluating the same frozen checkpoint across implementations isolates the effect of implementation differences from variability introduced by retraining. The results provide representative evidence that the assembled LDE and GLHF recipes retain comparable final optimization performance to the original implementations under matched evaluation conditions.

How are the baseline recipes constructed? We prioritize the original authors' released implementations as the reference for modularization. When these are unavailable, we follow the original papers and consult third-party implementations from established benchmarks such as MetaBox and PlatMetaX. If no public implementation is available, we reconstruct the method from its published algorithm description. We then express the method through task-specific components and assemble it into the shared training and evaluation workflow.

W2/Q2 (Consistency across independent meta-training runs).

How many independent training runs? The original RQ3 and RQ4 results used one meta-training run (seed 42) per learned setting. To further assess the consistency of these findings across independent meta-training runs, we have added two training seeds (43 and 44), giving three training runs per setting under the same budget of 100 episodes on eight training functions. Each checkpoint is evaluated on 16 held-out 40D BBOB functions with 30 test seeds per function. Appendices G.4 and H.4 report the results separately for each training seed.

RQ3: How consistent is the effect of configuration granularity? We retrained both Meta-Pop and Meta-Ind at all four population sizes. The table counts the functions on which Meta-Ind achieves lower mean final best-so-far fitness than Meta-Pop.

Population NNSeed 42Seed 43Seed 44
2014/1611/1615/16
5014/1611/1614/16
1006/167/167/16
2009/168/168/16

Meta-Ind retains its advantage on most functions at N=20N=20 and 5050 across all three training seeds. At N=100N=100, Meta-Pop instead performs better on 9–10 functions; at N=200N=200, the comparison is nearly even. These results support the population-size dependence reported in RQ3: individual-level control offers a consistent advantage at smaller populations, but this advantage does not extend uniformly to larger populations under the fixed training budget.

RQ4: How consistent are the state-representation findings? We independently retrained DDQN for all eight representations with DE, PSO, and ES at N=50N=50. The pretrained NeurELA encoder remains frozen. For each optimizer and training seed, the table shows the representation with the most function-level wins among the eight variants. A win denotes the lowest mean final best-so-far fitness; counts are out of 16 functions. F and P denote fitness-space and optimization-process information, respectively.

OptimizerSeed 42Seed 43Seed 44
DEF (13/16)F+P (15/16)P (13/16)
PSONeurELA (8/16)F+P (8/16)F+P (10/16)
ESF (7/16)F+P (9/16)F (12/16)

The leading representation changes across training seeds for all three optimizers. For PSO, F+P leads under both additional seeds, while NeurELA leads under the original seed. These results support the observation that representation effectiveness depends on the optimizer and further show sensitivity to training randomness. More broadly, a state design that performs well in one training run may not retain its advantage after retraining. Claims about its effectiveness should therefore be supported by comparisons across independent training runs. We thank the reviewer for this insightful suggestion, which prompted this analysis and helped clarify an evaluation issue of broader relevance to MetaBBO research.

Reviewer_hDzjOpenReview ↗

Follow-up Questions on W1

Thank you for the additional validation. However, reporting only the median of the absolute standardized differences removes their direction and hides variation across functions. Please provide the full per-function results, including signed differences, to show whether EvoX systematically outperforms or underperforms the original implementation and whether individual functions show substantial discrepancies.

Values such as 0.157 are also difficult to interpret without a reference for variation within the original implementation itself. I suggest comparing two independent batches of 30 runs from the original implementation using the same metric, repeating this to establish a reference distribution. The Original–EvoX comparison should follow the same sampling protocol. This would help assess whether cross-implementation differences exceed the variation typically observed when repeating experiments with the original implementation.

Reviewer_MDYuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents EvoX MetaBench, a composable framework and benchmark for meta-black-box optimization (MetaBBO). It defines interchangeable components for algorithm selection, algorithm configuration, solution manipulation, and algorithm generation, with shared training, validation, and test manifests over more than 11,000 problem instances and recipes. Built on EvoX, it batches compatible optimization runs on GPUs. The experiments compare representative methods, report CPU-to-GPU speedups for LDE and GLHF under selected workloads, and use controlled component studies to examine configuration, state representation, and generation operators.

Critical strengths
  1. The paper addresses an important methodological gap. MetaBBO methods are often released as coupled systems, making it difficult to tell whether a gain comes from the state features, controller, underlying optimizer, or training procedure. The task-specific interfaces and shared manifests make compatible components replaceable under common conditions. This could make controlled ablations and attribution more routine, while reducing duplicated implementation effort.

  2. The attention to computational scale is practically valuable. MetaBBO requires repeated optimization runs across instances and random seeds, and the framework uses EvoX for batched tensor execution. Figure 5 reports substantial CPU-to-GPU speedups for LDE and GLHF, including on an RTX 2080 Ti. Showing that selected workloads can be accelerated on this level of hardware makes the benchmark more attractive for researchers without large compute clusters.

Critical weaknesses
  1. My main concern is the unclear scope of GPU acceleration. Section 3.3 says components with compatible batched inputs and outputs can reuse tensorized execution, but the runtime study in Section 4.2 covers only LDE and GLHF. Table 4 also includes BBOPlace with external evaluators and HPO and neuroevolution through workflow adapters; their objective evaluations may not batch or accelerate in the same way. It is therefore unclear which suites and baseline recipes run end-to-end on GPU, which accelerate only selected stages, and which rely mainly on CPU or external calls. A support matrix and representative end-to-end timings across these execution categories would make the scalability claim easier to assess.

  2. My second concern is the missing total resource accounting. The paper specifies hardware and per-run evaluation budgets, Figure 5 gives per-episode or per-problem times, and Appendix E.1 caps algorithm-generation design at 136 API calls per target function. These details are useful, but they do not establish the total wall-clock time or CPU/GPU hours for the reported experiments, nor the aggregate LLM calls, tokens, or monetary cost. With more than 11,000 problem instances and some external evaluators, prospective users cannot readily estimate the cost of reproducing the results or running a broad benchmark sweep. A concise cost table by track would materially improve reproducibility and adoption.

Questions for authors
  1. For each problem suite and baseline family, which stages—objective evaluation, controller inference or training, and population updates—support batched GPU execution? What happens for external or adapter-based evaluators, and how do their end-to-end runtimes compare with CPU execution?
  2. Can the authors report total wall-clock time and CPU/GPU hours for the experiments, along with a cost estimate for a representative broad benchmark sweep? For the LLM-based baselines, how many calls and input/output tokens were used in aggregate, and what was their monetary cost or a price-independent equivalent?
4 comments

Thank you for the careful review and concrete suggestions. We provide point-by-point responses below.

W1/Q1 (Scope of GPU acceleration and end-to-end benefits).

We have added Appendix E ”Details of RQ1“, which covers the motivation and execution model (E.1), GPU and batching support (E.2), and stage-level and end-to-end runtime analysis (E.3).

Which combinations support GPU execution? Appendix E.2 provides separate support matrices for problem suites and method recipes. The problem matrix distinguishes GPU objective computation from batching across candidates and independent runs. The method matrix reports support for inference, learning, and population updates. A GPU-capable recipe paired with a compatible tensor evaluator can keep its numerical computation on the GPU. The execution conditions identify shape requirements, state handling, and host dependencies, including portfolio dispatch. LLM inference remains external, and execution of generated algorithms depends on their code.

Where do the speedups come from? The main text reports speedups during steady execution after warmup. Here we further examine the contributions of individual computational stages and, below, the time savings for complete processes. We compare LDE, GLHF, and LGA on 40D BBOB using an RTX 2080 Ti. Each LDE/GLHF training workload contains one batch of 16 trajectories, with an initial evaluation followed by 200 updates and the method's learning updates. LGA uses one ES iteration with 16 perturbations, each evaluated with this batch configuration. The following training-stage speedups at N=2000N=2000 come from one separate profile per device and configuration.

MethodInferenceLearningPopulationObjective
LDE (RL)6.06×6.88×98.45×10.38×
GLHF (SL)39.29×14.52×7.31×8.07×
LGA (NE)25.35×2.99×20.05×7.18×

LDE gains primarily from population operations, GLHF from inference and learning, and LGA from inference and population operations. Appendix E.3 also provides the corresponding stage runtimes and testing profiles.

What are the end-to-end benefits? Testing uses 30 trajectories, each with an initial evaluation and 199 updates. Workload time measures computation after warmup; process time additionally includes startup, initialization, warmup, and reporting. The following speedups are medians of five paired CPU/GPU ratios.

MethodNNTrain workloadTrain processTest workloadTest process
LDE501.30×0.91×1.44×0.96×
LDE200054.19×18.21×72.46×28.20×
GLHF500.80×0.95×4.76×1.26×
GLHF200021.73×16.65×36.70×26.61×
LGA501.38×1.24×1.60×0.95×
LGA200026.01×25.13×23.70×17.62×

Process overhead substantially reduces the gains for short workloads at N=50N=50. At N=2000N=2000, all three methods achieve substantial process-level acceleration in both training and testing.

How do adapters and external evaluators affect acceleration? We additionally evaluate LDE on BBOB, HPO-B, native HPO, neuroevolution, and small-scale BBOPlace. Native HPO uses an inner PSO-on-Sphere workflow; neuroevolution evaluates network parameters on digit images. The table reports median complete-process times and median paired speedups over five pairs for LDE.

SuiteNNCPU (s)GPU (s)Speedup
BBOB504.434.640.95×
HPO-B505.025.630.87×
Native HPO509.458.461.25×
Neuroevolution5012.475.822.33×
BBOB2000209.296.8330.64×
HPO-B2000206.709.4522.28×
Native HPO2000562.2826.1521.19×
Neuroevolution2000645.5120.5531.88×
BBOPlace103794.293938.480.96×

Native HPO and neuroevolution accelerate both objective evaluation and the surrounding optimizer; their objective stages achieve 27.19×27.19\times and 31.68×31.68\times speedups at N=2000N=2000. HPO-B retains CPU NumPy/XGBoost evaluation but benefits from GPU population operations. BBOPlace uses the same external CPU placement backend in both execution configurations; its evaluation cost dominates, so GPU execution provides little overall benefit. Including BBOPlace broadens MetaBBO evaluation to chip-placement tasks with expensive external objectives. Its adapter exposes the placement backend through a shared interface, so compatible MetaBBO methods can be compared on this application.

W2/Q2 (Resource accounting and benchmark reproduction cost).

We added Appendix J, "Training and Testing Costs," with resource budgets by track and suite, LLM usage, and practical study budgets. Note that, because of the addition of "Details of RQ1," the appendix you originally referred to as Appendix E has become Appendix F.

Training and testing costs. We recorded the RQ2 training and testing times on an RTX 2080 Ti as shown below. This covers ten learned methods and three classical optimizers on BBOB, plus ten methods on HPO-B, UAV, and LJ.

WorkloadCPU-hoursGPU-hours
AS training7.767.76
AC training4.464.47
SM training3.883.88
BBOB testing: 13 methods1.971.98
HPO-B testing2.632.63
UAV testing5.575.56
LJ testing0.150.15
Total26.4226.43

The RQ1 supplementary campaigns recorded 20.87 task-hours, 72.89 CPU-hours, and 5.38 GPU-hours; RQ5 searches recorded 4.37 task-hours. For three training seeds, the RQ3 and RQ4 budgets are 36.02 and 93.69 task-hours, respectively, based on historical timings and matched profiles on their original hardware.

LLM calls, tokens, and charges. Recorded usage and budget are summarized below; Appendix J additionally separates the four AG methods. Tokens are in millions and API charges are in USD.

WorkloadCallsInput (M)Output (M)USD
AG: 64 designs8,69232.46116.46312.678
AS-LLM2430.2020.0040.029
LAOS3,2006.4310.4101.206
OPRO1,9207.5620.8471.241
RQ5 operator studies1,9031.2000.7080.432
Recorded API usage15,95847.85618.43215.586

Here, we focus primarily on the most active AG area and its associated consumption. AG design and initial checks took 9.29 hours in parallel, or 481.42 cumulative task-hours, including API waiting and scheduling. Frozen-program testing took 0.81 hours, or 3.20 task-hours, and required no additional LLM calls.

Budget for a representative study. Comparing the four AC baselines, with one training run per method and thirty optimization seeds per test instance across held-out BBOB and all three application suites, requires about ten hours: 9.96 GPU-hours and 9.95 CPU-hours. For four AG baselines on sixteen targets, allow approximately 8,700 API calls and USD 12.68 with GLM-5.3-Flash at the recorded rates, plus local evaluation. Design and testing together take about ten hours under the recorded concurrency.

Dear Reviewer MDYu,

As the discussion phase comes to an end, please acknowledge the author's response and consider leaving them a comment on whether it has resolved some or all of your concerns.

Thanks!

Thank you for your detailed response. These responses resolved my concern, and I will keep my score of weak acceptance.

Comments on the paper

8 comments
Area_Chair_CE4KOpenReview ↗

Review by AC

Since we couldn't find emergency reviewers for this paper, I'm writing an additional review.

Paper Summary

This paper proposes EvoX MetaBench, which is a toolbox for benchmarking black-box optimization algorithms. EvoX MetaBench implements several benchmark problems, black-box optimization algorithms, and provides tools for evaluating large-scale benchmark runs. The authors showcase EvoX MetaBench in experiments, demonstrating the modularity of the proposed benchmarking suite.

Presentation: 2

Informativeness: 2

Soundness: 3

Strengths

Tools for cleanly benchmarking algorithms without confounding factors like differing state representations are valuable for the community. The proposed toolbox supports a wide range of algorithms and problem instances, making adoption by the community more likely.

The main argument for EvoX MetaBench is the decomposability of the benchmarking process and the design supports such decomposability.

Weaknesses

The idea behind EvoX MetaBench is not new and the authors acknowledge this. For instance, MetaBox-v2 provides many (although not all) of the features the proposed toolbox provides. It is not clear to me why already-existing projects cannot be extended to support more features. The goal should be to have one agreed-upon benchmarking tool for the community and it is not clear to me if adding new tools over existing efforts is the right way.

In general, the authors argue with decomposibility. While I see how the proposed toolbox achieves this, the authors do not show how previous works fall short of this goal.

Questions

What are the speedups at N=50-200?

In how far is MetaBox "coupled"?

Rating: 3 Confidence: 2

Thank you for taking the time to provide an emergency review despite your busy schedule, and for the constructive assessment and questions. All changes are highlighted in blue in the revised PDF. We provide point-by-point responses below.

Q1 (Speedups at population sizes N = 50–200).

The table below reports RQ1 speedups following the main-text timing protocol: mean CPU runtime divided by mean GPU runtime after warmup. Training uses batches of 16 trajectories; testing covers 30 runs on BBOB f4f_4. Appendix E presents a separate supplementary study, whose workload and process speedups are medians of five paired CPU/GPU runtime ratios.

MethodNNTraining speedupTesting speedup
LDE501.20×1.04×
LDE1001.56×1.31×
LDE2003.01×3.28×
GLHF502.36×2.18×
GLHF1002.48×3.57×
GLHF2003.93×8.76×

At N=50N=50, LDE shows limited gains, while GLHF achieves approximately twofold acceleration in both training and testing. The gains increase with population size for both methods. At N=200N=200, training speedups reach 3.013.01–3.93×3.93×, and testing speedups reach 3.283.28–8.76×8.76×.

W1/W2/Q2 (Component-level decomposition and the relationship to MetaBox).

Where is MetaBox coupled? MetaBox-v1 was developed for RL-based MetaBBO, where an RL policy adapts a low-level optimizer during search. MetaBox-v2 retains this agent–optimizer organization and provides reusable RL algorithms and shared training and evaluation workflows. These interfaces organize interactions between complete agents and optimizers, while observation construction, rewards, action granularity, and their connections to search operations remain defined within individual methods. Its SL- and ES-based baselines also implement their core learning logic internally: GLHF defines its networks, trajectory loss, and gradient updates, while LGA and LES implement parameter-search rollouts and performance aggregation. The coupling lies in the method-specific assumptions connecting these internal components, so changing one component can require coordinated adaptations elsewhere.

What does component substitution enable? We use LDE to illustrate this distinction. For a population of size NN, MetaBox-v2's LDE policy produces 2N2N values, which its optimizer interprets as NN individual-specific pairs of FF and CRCR. Replacing this policy with one that produces a single population-shared pair requires coordinated changes to the policy output, action handling during training and testing, and optimizer-facing parameter mapping. In EvoX MetaBench, this granularity change is specified by switching the adapter connecting the controller and optimizer from individual-level to population-level assignment. The RQ3 configuration-granularity study applies this assembly mechanism to separate the benefits of configuration diversity from those of learned control. This makes the changed design factor explicit and supports attribution of performance differences under shared experimental conditions.

Why build on EvoX rather than extend MetaBox? An extension offering the same capabilities would need to extract internal components into common interfaces, adapt method implementations to those interfaces, and coordinate tensor representations, independent run states, and evaluator integration for batched execution. These changes span method assembly and execution. We chose EvoX because it already provides tensorized optimizers, problem interfaces, and batched execution primitives. Building the component layer on this foundation lets us reuse that infrastructure and design component substitution together with scalable execution.

How do we limit fragmentation? We reuse EvoX optimizers and external evaluation backends through adapters. For BBOB, we follow MetaBox-v2's training/test function partition and record instance choices in shared manifests. These choices reduce duplicated infrastructure and support comparisons based on established benchmark conventions. We will also continue extending the platform with reusable plugins, including representation-learning modules that can serve multiple compatible methods. This development direction allows new functionality to be shared across method assemblies, beyond adding individual baselines.

We have added Appendix B.5, “Component-Level Comparison with MetaBox,” and a schematic illustrating component organization and substitution through the configuration-granularity example.

Reviewer_hDzjOpenReview ↗

Follow-up Questions on Behavioral and Performance Fidelity of Modularized Implementations

Thank you for clarifying the component-level differences from MetaBox. This connects to my Concern 1 regarding the fidelity of the reimplemented baselines. Some coupling may preserve intentional co-design between components, while modularization facilitates reuse and controlled component substitution. How do you demonstrate that EvoX MetaBench gains these benefits while preserving the original methods’ behavior and performance, rather than achieving flexibility at the expense of implementation fidelity? This motivates two related questions:

(i) Does the modularized implementation preserve the original method’s behavior? When extracting components, did you identify dependencies arising from their joint design or tuning? How are these dependencies preserved, and how are coordinated adjustments handled when the components are assembled together? A concrete example would help.

(ii) Does the modularized implementation achieve performance comparable to the original implementation? In your response to my review, you report the median across functions of the absolute mean performance difference normalized by the pooled standard deviation. This removes the direction of the differences and hides variation across functions. Please provide the full per-function results, including signed differences, to show whether EvoX systematically outperforms or underperforms the original implementation. Values such as 0.157 are also difficult to interpret without a reference for variation within the original implementation itself. I suggest comparing two independent batches of 30 runs from the original implementation using the same metric, repeating this to establish a reference distribution. The Original–EvoX comparison should follow the same sampling protocol. This would help assess whether cross-implementation differences exceed the variation typically observed when repeating experiments with the original implementation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.