acceptodds
Under review as a conference paper at ICLR 2027

JANUS: Online Jacobian-Aligned Infill for Black-Box Optimization

Abstract

Population optimizers such as CMA-ES, DE, and multi-objective evolutionary algorithms drive search mainly through selection signals that are scalar or rank based: such a signal indicates that one candidate outperforms another, but not the local direction responsible for the improvement. JANUS (Jacobian-Aligned Newton-Unified Search) is a plug-and-play infill module that extracts this missing local geometric signal without replacing the host optimizer. It estimates a local Jacobian from the recent evaluation trace; the same Jacobian yields both a damped Gauss–Newton exploitation candidate and a trace-preserving exploration metric, reserving a fraction of the host's per-generation candidate slots for geometry-guided infill rather than spending evaluations on top of the host's budget. Unlike MetaBBO methods, JANUS needs no offline training or task distribution, estimating this geometry on the fly from the current run alone, while the host keeps full control of selection, survival, covariance adaptation, and step-size control. Under same-protocol comparisons, JANUS improves the CMA-ES host on 11–15/16 BBOB functions across . It also attains the best mean error on 13 of the 16 functions at in the complete NN-BBO/MetaBBO baseline comparison, with no training cost, and yields a geometric-mean improvement over the host on a BBOB subset. On structured and multi-objective tasks, JANUS gives the best mean cost on 1135-dimensional UAV path planning ( vs. the strongest baseline), and it improves SMS-EMOA/AGE-MOEA2 hosts on 12/38 multi-objective tasks with zero significant regressions.

Primary area:
optimization
Keywords:
Jacobian-Aligned, BBO

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oHo3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces JANUS, a candidate-generation module used for black-box optimization. JANUS uses historical evaluation data to fit a local model, which is simultaneously employed for the exploration and development of candidate solutions. These candidate solutions will be further integrated into the original optimizer. The authors claim that JANUS can improve the search performance while maintaining the budget unchanged.

Critical strengths
  1. The module proposed in this article is highly flexible and can be seamlessly integrated into any optimizer
  2. The experimental results appear to be very promising.
Critical weaknesses
  1. This paper is hard to follow in several places. For instance, in the abstract, too many technical terms were piled up at the very beginning, without clearly informing the readers what problems these mathematical tools solved and how the contributions should better highlight the innovativeness of the proposed method.
  2. The application of the proposed JANUE in different scenarios does not seem to fully align with its core methodological design. For example, in the MOO setting, only the Jacobian matrix of the objective vectors is used to construct the exploration metric, without simultaneously guiding both exploitation and exploration as emphasized in the main methodology. The authors should clarify how these scenario-specific implementations support the claimed unified framework.
  3. The generalizability of JANUS across different optimization algorithms has not been fully verified. Although the authors have conducted experiments on optimizers including CMA-ES and DE, the performance of this framework when integrated with PSO, SHADE, and the more representative MetaBBO method has not been systematically evaluated.
Questions for authors

See weaknesses above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer oHo3 (Part 1)

W1. Clarity, and what is new

We rewrote the abstract, introduction and method overview.

  • The abstract now opens with the problem (balancing local improvement and exploration under a fixed evaluation budget), states the idea in one sentence (one model fitted to recent evaluations is reused to propose improvement steps and to shape exploratory sampling), and then gives the main results, led by the matched Sep-CMA-ES comparison and its confirmation on new optimizer seeds. Technical constructions such as the residual Jacobian, damping and trace normalization are developed in Sec. 3, where they are needed.
  • The introduction poses two questions: how one fit can serve both a point proposal and an exploration distribution, and how the influence of a possibly wrong model can be bounded. The three contributions answer them: one fit for two roles; plug-and-play adapters that keep the host's batch, budget, selection and adaptation; and a controlled evaluation that includes where the method does not help.
  • Sec. 3 opens with a map of its parts and defines the two ways infill reaches a host: candidate mode (infill replaces native candidates; all benchmarks) and covariance mode (the fitted geometry is blended into the sampling covariance; AWB only).

The novelty relative to prior model-assisted and learned optimizers is summarized in Table 1:

MethodNo offline pretrainingNo online NN fittingValue-only feedbackOne fit for step and samplingHost-slot insertion
GLEET / LES×✓✓××
Guided ES✓✓×××
TuRBO✓✓✓××
EvoGrad2✓×✓××
ABOM✓×✓××
JANUS✓✓✓✓✓

W2. Scenario-specific implementations versus the unified framework (MOO uses only exploration)

The reviewer is right that the MOO adapter uses only one of the two roles. The revision states why and no longer presents MOO as an instance of joint exploitation and exploration.

What is common to all scenarios is the pipeline: fit a local model to recent evaluations, turn the fitted geometry into proposals, and let the host evaluate and select. Which roles are active follows from the feedback (Sec. 3.2, Table 11):

FeedbackExploitation stepMetric-shaped explorationHost keeps
Observed nonnegative components (AWB)Yes, Gauss–NewtonYesSelection and adaptation
Scalar objective (BBOB, UAV)Yes, fitted-gradient stepYes, when ρ > 0Selection and adaptation
JANUS-GN (ρ = 0)YesNoSelection and adaptation
Objective vector (MOO)NoYesPareto survival

A vector objective does not define a single descent target without a preference rule or scalarization, so the MOO adapters propose no exploitation step and leave trade-offs to the host's Pareto survival.

We also report what this exploration-only use achieves. On 18 low-objective and 21 many-objective tasks (5% infill, ten seeds, 20,000 FEs), fixed-reference hypervolume improves significantly over no infill on 1/18 and 2/21 tasks after Holm correction and is not distinguished from matched random infill (Sec. 5.4, Table 40). The same holds for metric-only infill on a DE host (Table B). Separately, in tuned 500D CMA configurations the fitted metric's orientation does not consistently outperform a rotated-metric control; that control changes both roles and so isolates neither (App. G.4). The paper's supported claims therefore concern the scalar and residual settings, where both roles are active; the conclusion states that metric-shaped exploration used alone is not shown to outperform random infill. The earlier MOO rank summaries are retained in App. C.4 and labeled as descriptive results of an earlier study with unadjusted tests.

AuthorsOpenReview ↗

Response to Reviewer oHo3 (Part 2)

W3. Generalization to PSO, SHADE and MetaBBO hosts

PSO and SHADE are now tested, together with MadDE, DE and jDE21:

JANUS adaptervs native host, 100Dvs native host, 500Dvs matched random infill, 100Dvs matched random infill, 500D
SHADE (scalar model)9/012/29/012/0
PSO (scalar model)11/013/05/09/1

16 functions, 30 seeds, 20,000 FEs, paired Wilcoxon tests with Holm correction per contrast and dimension; the full host set is in Table B and manuscript Table 3, and adapter settings are in Table 35. The PSO configuration was selected on seeds 201–205 of the same 16 functions and evaluated on seeds 1–30, so its result concerns new seeds, not unseen functions. For PSO, infill replaces selected particle positions after the ordinary velocity and position update. For SHADE, infill is excluded from the native F/CR success-memory credit because those parameters did not generate it (App. A.3, App. G.3).

A MetaBBO method as the receiving host is not tested. Such an adapter must define how externally generated candidates enter the learned policy's state and credit assignment, which differs for each method. MetaBBO methods appear in the paper as external baselines (Tables 15 and 20), and integration into a learned optimizer is listed as future work in the conclusion.

Thanks for the author's response. I am sticking to my initial judgment and keeping the score unchanged.

AuthorsOpenReview ↗

Response to Reviewer oHo3

Thank you for taking the time to consider our revision. We appreciated your suggestion to evaluate JANUS with PSO, SHADE, and MetaBBO as hosts. It prompted us to add the PSO and SHADE experiments in Table 3, where we compare JANUS with each native host and with matched random infill.

If you have a moment, we would be grateful to know which concern you feel we have yet to address adequately. Even a brief comment would help us better understand your assessment and where the paper still needs work.

Reviewer_eyA9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an online Jacobian-aligned infill method for black-box optimization. The candidate-generation interface uses local fits for exploitation in scalar minimization and metric-shaped exploration, while preserving the host’s update rules and evaluation budget. The test results on BBOB, UAV path planning, multi-objective optimization, and AWB calibration show favorable performance in several settings. Nevertheless, implementation inconsistencies, incomplete statistical evidence and inadequate baseline algorithms weaken the empirical support.

Critical strengths
  1. The method can be readily integrated with existing population optimizers, since it offers a simple candidate-generation interface that preserves the host's update rules and requires no offline training.

  2. Within a strict black-box evaluation budget, the evaluations already performed by the host can be reused to further improve the quality of the candidate solutions without incurring any additional objective evaluations.

  3. The proposed candidate-generation interface has a clear structure: it uses a local fit to generate a model-based descent proposal together with metric-shaped exploration samples, after which the host evaluates the candidates and carries out its routine population update.

  4. The paper covers a variety of search dimensions, host optimizers, and task types, and it reports promising results across several experimental settings.

Critical weaknesses
  1. Algorithm 1 does not specify how native candidate slots are selected for replacement or analyze how different replacement rules affect optimization performance.

  2. The fixed allocation of one exploitation proposal in scalar minimization lacks sufficient justification. Its suitability under different levels of fit reliability has not been established.

  3. The choice of fixed model-refresh intervals is insufficiently justified. The effects of refresh frequency on optimization performance and computational cost are not adequately analyzed.

  4. The most recent 300 evaluations may not form a suitable local fitting window. The reported sensitivity sweep does not sufficiently justify this setting across search dimensions and search stages.

  5. Shared fitting has not shown a consistent advantage over separate fits. Without a direct runtime comparison, its benefits in optimization performance and computational cost remain unclear.

  6. The original BBOB/MOO adapters and run-level records are unavailable, and the released sampler differs from Algorithm 1. Some main benchmark results therefore cannot be verified against the current candidate-generation interface.

  7. The 500D BBOB comparisons use different numbers of runs and mainly rely on tabulated means, without confidence intervals or statistical tests to assess the reliability of the observed differences.

  8. The MOO interface does not specify normalization or weighting of the objective vector. The local metric GG depends on objective scales, but the paper provides no corresponding sensitivity analysis.

  9. A 300-point window leaves directions unidentified in high-dimensional linear residual regression despite ridge regularization. The impact of this limitation on the exploitation proposal and metric-shaped exploration remains insufficiently evaluated in the benchmark tasks.

  10. The baseline methods are insufficient and not up-to-date.

Questions for authors
  1. How does Algorithm 1 select native candidate slots for replacement? If selection is random, could the local model guide replacement of less informative candidates without increasing the evaluation budget?

  2. Why allocate exactly one exploitation proposal in scalar minimization? Could reliable fits justify more proposals and unreliable fits justify skipping exploitation? Please provide comparisons across different proposal counts.

  3. What justifies fixed model-refresh intervals? How do they affect optimization performance and computational cost, and could refreshes be triggered by fit quality or changes in the host distribution?

  4. Why use the most recent 300 evaluations rather than select the fitting window adaptively based on sample distance or fit quality? How does the reported sensitivity sweep on 100D BBOB justify this choice at other dimensions?

  5. What is the main benefit of shared fitting when it does not consistently outperform separate fits? Please directly compare their runtimes and analyze the tradeoff between optimization performance and computational cost.

  6. With the original BBOB/MOO adapters and run-level records unavailable, can the authors provide reproducible benchmark experiments consistent with Algorithm 1? Which main performance claims are supported by the current implementation?

  7. Why do the 500D BBOB comparisons use 30 runs for JANUS-GN but 10 for full JANUS and external baselines? Please explain this choice and provide confidence intervals and statistical tests for the reported differences.

  8. How does the MOO interface handle objective scales when constructing G=JTJ? Are individual objectives normalized or weighted? Could the authors assess sensitivity by rescaling each objective by a positive factor?

  9. How does limited Jacobian identifiability affect the exploitation proposal and metric-shaped exploration in high-dimensional benchmark tasks? Can the authors provide diagnostics beyond the controlled affine residual maps to assess this impact?

  10. The comparison should include a broader set of recent state-of-the-art black-box optimization methods.

7 comments
AuthorsOpenReview ↗

Response to Reviewer eyA9 (Part 1)

We answer the ten questions in order; each answer also covers the weakness with the same number. Q1–Q4 share one set of one-factor controls, shown first.

Table C. One-factor design controls (manuscript Table 6; run times Table 36).

FactorChanged setting vs. referenceW/L over 8 settingsTotal run time, 100D / 500D (s)
Referencefixed slots, 1 exploitation proposal, h = 4, n = 300, ρ = 0.2—4.12 / 21.50
Replacement slotsuniform random vs. fixed0/0—
Exploitation proposals0 vs. 10/4—
Exploitation proposals2 vs. 11/0—
Refresh intervalh = 1 vs. 41/17.49 / 31.72
Refresh intervalh = 8 vs. 40/23.74 / 20.17
Refresh intervalh = 16 vs. 40/23.34 / 20.05
Windown = 100 vs. 3000/03.38 / 20.18
Windown = 600 vs. 3001/04.69 / 30.50
Windown = 1,200 vs. 3000/06.91 / 52.55

Scalar-CMA reference configuration on a canonical CMA-ES host; eight settings (f6, f10, f22, f23 at 100D and 500D), ten seeds, 20,000 FEs; W/L uses paired two-sided Wilcoxon tests with Holm families of 8 (slots), 16 (count), 24 (refresh) and 24 (window) tests. The ten-seed controls have limited inferential resolution under these Holm families, so 0/0 entries should not be read as equivalence. Run times are medians over the same 40 runs per entry (four functions, seeds 1–10) on CPU, including diagnostic instrumentation.

Q1. How are replacement slots selected?

By a fixed adapter rule, not at random. The CMA adapter places the exploitation proposal in the first slot and exploration samples in distinct tail slots; the DE adapter preserves target–offspring associations; MOO adapters use distinct offspring slots. Sec. 3.1 and Algorithm 1 (lines 6–12) now state this.

The choice of rule has no detected effect: fixed versus uniform-random distinct slots gives 0/0 (Table C). This is expected for a CMA-type host, whose native candidates are exchangeable draws before evaluation.

Using the model to choose which native candidates to overwrite is possible without extra evaluations, since it only requires scoring native candidates on the fitted model. We have not tested it. It would add a model-based pre-selection step whose benefit depends on the same prediction quality examined in Q9, and we list it with reliability-based allocation as future work.

Q2. Why exactly one exploitation proposal?

One proposal is a conservative default: it gives the fitted direction one trial per scheduled generation and leaves the rest of the batch to the host. The controls support having it and suggest it is not always optimal. Removing it regresses four of eight settings; adding a second improves one setting with no detected regression (Table C). The second proposal is another step length from {0.25, 0.5, 1} along the same fitted direction, so this control does not test independent additional directions.

On adapting the count to fit reliability: the recorded data indicate that held-out prediction error alone is a weak signal for skipping exploitation. Grouping 13,680 refreshes of the 500D runs by diagnostic NMSE (Table 39), the median exploitation improvement rate is similar within a function whether NMSE is below or above one:

FunctionNMSE < 1NMSE ≥ 1
f620.0%22.2%
f1052.4%48.8%
f2260.0%50.0%
f23no such refreshes0.0%

The pooled gap (51.4% versus 23.0%) largely reflects which functions fall in each group. Skipping exploitation whenever NMSE ≥ 1 would discard useful proposals on f6 and f10; f23 is the case where skipping would cost nothing. An exploratory fit-quality gate, added after seeing diagnostics and therefore not a confirmatory result, gave 0/0 in a short-budget check (App. G.3). A reliability rule for choosing 0, 1 or 2 proposals remains future work, and the conclusion now says so.

Q3. Why fixed refresh intervals, and what do they cost?

A fixed interval makes fitting cost and candidate allocation predictable. In this adapter fitting and injection occur together, so the interval controls both.

The measured trade-off (Table C): refreshing every generation costs 1.5–1.8 times the reference run time and gives one improvement and one regression; refreshing every 8 or 16 generations saves 6–19% of run time and regresses in two of eight settings each. The reference h = 4 is thus a reasonable middle setting on this subset; the Sep-CMA-ES configurations use h = 2, chosen by the selection protocol in App. G.1.

Refreshes triggered by fit quality or by changes in the host distribution are not implemented. The only related mechanism we tried is the exploratory gate mentioned in Q2, which skips injection after a scheduled fit and leaves the schedule unchanged.

AuthorsOpenReview ↗

Response to Reviewer eyA9 (Part 2)

Q4. Why the most recent 300 evaluations?

It is a simple fixed-cost rule that follows the current search region and needs no distance or quality threshold. The original 100D sweep does not by itself justify the choice at other dimensions, so we extended the window control to 500D (Table C; App. B.1 now states this limitation of the sweep). Across 100D and 500D, windows of 100 and 1,200 are not distinguished from 300, and 600 improves one of eight settings; a 1,200-point window costs 1.7–2.4 times the reference run time.

The window is not universally 300: the selected JANUS-GN configuration on Sep-CMA-ES uses 1,200, at 6.4 times the host's run time against 1.8 times for the 300-point two-role configuration (App. G.7). Those two configurations differ in both window and exploration allocation, so only Table C isolates the window. Adaptive selection by sample distance or fit quality is untested.

A window of 300 at d = 500 leaves at least 201 directions unidentified by a linear residual fit (Sec. 4); Q9 addresses the consequences.

Q5. What is the benefit of shared fitting?

The benefit is computational. We do not claim an optimization benefit from sharing, and the revised text says so (Sec. 5.5).

  • With identical inputs, one fit and two duplicate fits produce identical proposals, and sharing reduces kernel time by 43.9–49.3% (Table 8a):
DimensionOne shared fit (ms)Two duplicate fits (ms)Reduction
300.3040.54343.9%
1001.0431.99147.6%
5006.76913.35849.3%
100015.42527.83144.6%
  • When the two roles use independently reweighted fits instead (Shared-B versus Separate-B; four functions, ten seeds, 20,000 FEs), sharing lowers accumulated fitting time by 50.6% at 100D and 41.6% at 500D, and total optimizer time by 10.4% (3.49 vs. 3.89 s) and 6.2% (18.86 vs. 20.10 s) (Table 8b). These trajectories can differ, so the ratios are descriptive.
  • Final error: the reweighted shared and separate fits show no detected difference on the diagnostic subset (paired Wilcoxon, one Holm family of 24 tests; App. G.4). The earlier 288-run study (three functions, eight seeds, 3,000 FEs) is descriptive only: with eight seeds and Holm correction over 24 contrasts, no contrast can reach adjusted p < 0.05 (the attainable and observed minimum is 0.1875), a limitation we now state explicitly (App. D).

End-to-end cost relative to each host is in App. G.7: 1.8× for the two-role Sep-CMA-ES configuration, 1.25–1.40× for the scalar-CMA reference configuration, and 1.22–1.46× for the CUDA-accelerated MOO adapters.

AuthorsOpenReview ↗

Response to Reviewer eyA9 (Part 3)

Q6. Reproducible experiments consistent with Algorithm 1; which claims rest on the current implementation?

Yes. All new paired experiments were run with a packaged candidate-insertion runner that implements Algorithm 1, with per-run records. App. A.1 and Table 10 now state, study by study, which implementation and which records support each result:

ResultImplementationRecords
Sep-CMA-ES at 500D: evaluation and confirmation cohorts (Table 2, Tables 30–33)Candidate-insertion runner (Algorithm 1)Per-run configurations, seeds, problem and initial-state fingerprints, source hashes, terminal values
Host transfer: jDE21, DE, SHADE, MadDE, PSO (Tables 3, 34)Same runner with host adaptersPer-run records
Design controls, model diagnostics, sharing run times (Tables 6–8, 36, 38–39)Same runnerPer-run and per-refresh records
Fixed-reference MOO study, 39 tasks (Table 40)Normalized vector-metric adapterPer-run records
500D external-baseline table (Tables 4, 15, 16)Earlier implementation (sources archived); CMA-ES is a named baseline, not a matched hostStored ten-run terminal values for JANUS, JANUS-GN, CMA-ES and random search; external baselines as summary means
30D/100D BBOB table (Table 14)Earlier implementationRun arrays archived, but not mapped to every table cell; reported as tabulated summaries on the 15 retained functions, with the untraceable f20 entries omitted at both dimensions
MOO rank summaries (Tables 22–23)Earlier implementationRun-level hypervolume values for every task and ranked method (1,620 and 2,000 values); all average ranks and normalized-HV summaries reproduced by re-aggregation
UAV (Tables 5, 19–21)Earlier implementation (configuration and sources archived)All run values at d = 30; all 28 per-terrain tests recomputed
AWB (Tables 9, 26)Component sampler, covariance modeIndexed trajectory of one run per method; recorded scene losses

The headline within-host claims (Table A and Table B above) therefore rest entirely on the current implementation. The external-baseline, UAV and MOO rank results come from the earlier implementation and are labeled as such. Its sources are archived, though run-time version hashes are missing for some executions. Significance statements about these results are limited to the ten-run 500D tests in Table 16 and the UAV per-terrain tests in Table 20; both compare archived implementations and are not matched-host ablations. The MOO rank annotations remain unadjusted and descriptive.

On the difference between the released sampler and Algorithm 1: Algorithm 1 is the scheduled candidate-insertion variant, which fits and injects only at scheduled generations, and it is what the paired studies run. The component sampler used for AWB supports covariance mixing and may reuse a cached metric between Jacobian refreshes; App. A.1 describes that difference.

The compact supplement includes the frozen-configuration confirmation records, the stage-aligned diagnostic aggregates used for Q9, the archived 500D terminal arrays and the scalar-reference diagnostic records, with offline verification commands. It also provides the run-level extracts and scripts used for the statistical checks described here. Its manifest and claim map specify the included evidence; it is not the complete archive of the broader campaign. From the supplement root (JANUS_Rebuttal):

python reproducibility/check_package.py --new-only
python reproducibility/verify_confirmation.py --out-dir verified_new
python historical_500d/recompute_statistics.py
python reference_diagnostics/verify_results.py
python author_verification/verify_second_pass.py --out verified_second_pass.json

None of these commands evaluates an objective. The first checks the file manifests. The second checks the confirmation cohort (1,440 runs; all 32 function-level tests, giving 13/0 and 13/1) and the 120 aligned diagnostic run aggregates (13,680 refreshes). The third recomputes the 48 ten-run 500D tests from 640 stored values. The fourth checks the 288 runs and 24 contrasts of the scalar-reference study. The fifth recomputes the 72 design-control tests and the 64 DE metric-infill tests from 2,880 terminal records.

Reviewer_xYvxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a candidate-generation module that fits a local model to the host’s existing evaluations and uses it to replace native candidates. The authors introduce a fitted residual Jacobian supplies model-based descent proposal and metric-shaped exploration samples. An objective gradient and a positive semidefinite metric are obtained from the scalar fit. Clipping, covariance trace normalization, and limited candidate allocation are used to control the proposal scale and budget. Experimental results show that the proposed method demonstrates superior performance on BBOB, UAV path planning and multi-objective optimization benchmark.

Critical strengths
  1. JANUS replaces selected candidate slots, keeps the host’s evaluation budget and update rules, and can be adapted to CMA-style and DE hosts.

  2. This paper provides clearly stated theoretical properties.

  3. The experimental results show competitive performance across multiple domains. The evaluation conprehensively covers BBOB, UAV, MOO and AWB.

Critical weaknesses
  1. The compared baselines are limited. Besides vanilla CMA-ES and DE, comparing JANUS and JANUS-DE with advanced CMA-ES and DE variants such as SEP-CMA-ES, IPOP-CMA-ES, MadDE, jDE21 could enhance the comprehensiveness of the evaluation.

  2. In the experiment, only mean final errors are reported. The standard deviation of the errors are omitted. Wilcoxon signed-rank test could be applied to enhance the convincingness of the results.

Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer xYvx

W1. Baselines: Sep-CMA-ES, IPOP-CMA-ES, MadDE, jDE21

All four requested variants are now compared. Three are receiving hosts with paired tests; IPOP-CMA-ES is compared as a baseline, but it triggers no restart at this budget and so coincides with active CMA-ES.

  • Sep-CMA-ES (500D). Both configurations improve significantly on 13 of 16 functions; JANUS-GN has no detected regression and JANUS has one (f22, 0.1% or less). The counts repeat on new optimizer seeds (Table A above; manuscript Table 2, App. G.1–G.2).
  • MadDE. 8/1 at 100D and 12/1 at 500D against native MadDE; 11/0 and 12/0 against matched random infill (Table B; manuscript Table 3). MadDE uses its documented population cap of 200 for this budget.
  • jDE21. 8/0 at 100D and 11/0 at 500D against native jDE21 with metric infill. This contrast has no random-infill control, and the analogous metric-only infill on DE is not distinguished from random infill (0/0, 0/3), so we do not attribute the jDE21 gain to the fitted metric (App. G.3). Both jDE21 variants execute 20,010 calls; final errors use only the first 20,000.
  • IPOP-CMA-ES and active CMA-ES. At 500D the default scalar-CMA configuration of JANUS gives 14/0 against IPOP-CMA-ES, 14/0 against active CMA-ES and 14/0 against its own canonical CMA-ES host (30 paired seeds; Holm families of 32 tests over 100D and 500D). In the tested IPOP setting no restart is triggered within 20,000 FEs: all 480 active/IPOP run pairs agree in terminal error and evaluation count. The IPOP comparison therefore adds no evidence beyond active CMA-ES, and JANUS was not attached to a restarting host. Within-host on active CMA-ES, a tuned 500D configuration gives 15/0 against its active host, while a selected 100D configuration gives 0/0 against its active host and 5/0 against Sep-CMA-ES (App. G.3).

We also report the less favorable CMA-type results below 500D (1/5 at 100D and 2/2 at 30D for the default configuration on canonical CMA-ES; 2/5 for a selected exploitation-only configuration on Sep-CMA-ES at 100D) in Sec. 5.1 and App. G.3.

W2. Standard deviations and Wilcoxon signed-rank tests

Both are now reported.

  • Paired host studies. Two-sided Wilcoxon signed-rank tests with Holm correction are used for the Sep-CMA-ES evaluation and confirmation cohorts (Tables 30 and 32), with means, sample SDs and paired-bootstrap intervals (Tables 31 and 33). The host-transfer study (Tables 3 and 34) uses the same paired test with Holm correction per contrast and dimension. For example, in the confirmation cohort:
FunctionSep-CMA-ES mean ± SDJANUS-GN mean ± SDJANUS mean ± SD95% CI, median log10(JANUS / Sep-CMA-ES)
f123004.9 ± 112039.92 ± 29.14139.42 ± 253.4[−2.105, −1.402]
f1375.72 ± 10.378.49 ± 1.2617.75 ± 2.05[−0.668, −0.594]

Values from Table 33: 30 seeds; SD with ddof = 1; paired seed bootstrap with 10,000 replicates. Intervals are per function and not simultaneous.

  • External-baseline tables. The 30D/100D table already reported mean ± SD (Table 14). For the 500D table we now report SDs and bootstrap intervals from the stored ten-run values (Table 16). Pairing is not available for these runs, so we use two-sided Mann–Whitney tests with Holm correction over 16 functions instead of the signed-rank test: 15/0 for JANUS against CMA-ES, 15/0 for JANUS-GN against CMA-ES, and 9/1 for JANUS against JANUS-GN. The external learned baselines are available as summary means only and remain descriptive.
  • UAV. Mann–Whitney tests with Holm correction across the 28 terrains are reported for every baseline (Table 20): for example 24/3/1 (improvements / no detected difference / losses) against CMA-ES and 22/5/1 against DE.

Thanks for the detailed responses and additional experimental results. The response address my concern. I am keeping my weak acceptance recommendation.

Thank you for taking the time to review the additional experiments and our response. We’re glad the revision addressed your concern.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (Part 2)

2. Scope of the results

  • Dimension. With the default scalar configuration on canonical CMA-ES, JANUS recorded 14 significant improvements and no regressions at 500D, one improvement and five regressions at 100D, and two of each at 30D. The clearest gains for this configuration occur at 500D (Sec. 5.1).
  • Evaluation budget. On UAV path planning, JANUS reduced mean cost relative to CMA-ES by 33.7% at 2,500 evaluations, 1.2% at 10,000, and 0.05% at 20,000. The advantage was largest at the shortest budget tested (Table 19).
  • Multi-objective optimization. The adapter uses the fitted metric for exploration and leaves Pareto selection to the host. Fixed-reference hypervolume improved significantly over the native host on 3 of 39 tasks. The comparison with matched random infill detected no significant difference (Table 40).
  • Metric orientation. In the tuned 500D CMA experiments, the fitted orientation showed no consistent advantage over a rotated control. The rotation acts on both the exploitation step and the exploration distribution, so this experiment assesses their joint use of the fitted geometry (App. G.4).

3. Corrections

Each mean in Tables 4 and 15 now uses ten runs per function. For JANUS-GN, we used the first ten runs from its separate 30-run series. Table 16 reports standard deviations for JANUS, JANUS-GN, and CMA-ES, along with 95% bootstrap confidence intervals and Holm-adjusted Mann–Whitney tests for their pairwise comparisons.

AuthorsOpenReview ↗

General response (Part 1)

We thank the reviewers for their careful reading and constructive feedback. The revision adds paired experiments on several host optimizers, including PSO and SHADE, and a confirmation study on Sep-CMA-ES with new optimizer seeds. We also revised the abstract and introduction to explain the central idea of JANUS: a model fitted to recent evaluations can guide both local improvement and exploratory sampling. Section 3 describes how this guidance enters the host optimizer through candidate replacement in the benchmark studies or covariance mixing in the AWB application. Table 1 compares how JANUS and related methods use feedback and generate candidates.

1. Paired comparisons on host optimizers

We compare each host with and without JANUS on the same problem instance, with the same optimizer seed and evaluation budget. Where available, a matched random-infill control replaces the same fraction of candidates using isotropic perturbations around the same elite points. The comparison with the native host measures the benefit of adding JANUS. The random control tests how much the fitted proposals contribute beyond candidate replacement itself.

Table A. JANUS on Sep-CMA-ES at 500D (manuscript Table 2). W/L counts functions with a significant improvement or regression relative to the native host. Each cohort uses 16 BBOB functions, 30 paired optimizer seeds, and 20,000 evaluations per run. We use two-sided paired Wilcoxon tests with Holm correction across the 16 functions.

ConfigurationW/L, evaluation seeds 1–30W/L, confirmation seeds 301–330f12 median error, confirmationf13 median error, confirmation
Sep-CMA-ES——2,786.574.8
JANUS-GN, exploitation only (n = 1,200, ρ = 0)13/013/037.48.3
JANUS, both roles (n = 300, ρ = 0.2)13/113/133.918.0

We selected the two configurations on eight other functions and froze them before confirmation. The confirmation repeated the comparison on the same 16 functions and problem instances with new optimizer seeds. Both configurations reproduced their original W/L counts. The size of the effect varies by function: the median error fell substantially on f12 and f13, while the single significant regression for the two-role configuration, on f22, increased the median by at most 0.1%. Tables 30–33 report the per-function results.

Table B. JANUS on additional hosts (manuscript Tables 3 and 34). Each entry gives W/L over 16 functions with 30 seeds and a budget of 20,000 evaluations per run.

JANUS adapterNative host, 100DNative host, 500DMatched random infill, 100DMatched random infill, 500D
jDE21, metric infill8/011/0Not testedNot tested
DE, metric infill10/013/00/00/3
DE, scalar model14/015/013/014/0
SHADE, scalar model9/012/29/012/0
MadDE, scalar model8/112/111/012/0
PSO, scalar model11/013/05/09/1

The scalar-model adapters recorded more improvements than regressions against both controls at both dimensions. The two DE adapters illustrate why the random control matters. At 500D, metric-only infill improved on native DE on 13 functions but lost to matched random infill on three, with no significant wins. The scalar-model adapter improved on matched random infill on 14 functions.

For the scalar-model rows, Holm correction covers 16 functions per contrast and dimension. For jDE21 and metric-only DE, it covers 32 tests per contrast across both dimensions. The PSO configuration was selected and evaluated on separate optimizer seeds for the same 16 functions. Table 35 gives the adapter settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.