- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces JANUS, a candidate-generation module used for black-box optimization. JANUS uses historical evaluation data to fit a local model, which is simultaneously employed for the exploration and development of candidate solutions. These candidate solutions will be further integrated into the original optimizer. The authors claim that JANUS can improve the search performance while maintaining the budget unchanged.
Critical strengths
- The module proposed in this article is highly flexible and can be seamlessly integrated into any optimizer
- The experimental results appear to be very promising.
Critical weaknesses
- This paper is hard to follow in several places. For instance, in the abstract, too many technical terms were piled up at the very beginning, without clearly informing the readers what problems these mathematical tools solved and how the contributions should better highlight the innovativeness of the proposed method.
- The application of the proposed JANUE in different scenarios does not seem to fully align with its core methodological design. For example, in the MOO setting, only the Jacobian matrix of the objective vectors is used to construct the exploration metric, without simultaneously guiding both exploitation and exploration as emphasized in the main methodology. The authors should clarify how these scenario-specific implementations support the claimed unified framework.
- The generalizability of JANUS across different optimization algorithms has not been fully verified. Although the authors have conducted experiments on optimizers including CMA-ES and DE, the performance of this framework when integrated with PSO, SHADE, and the more representative MetaBBO method has not been systematically evaluated.
Questions for authors
See weaknesses above.
4 comments
Response to Reviewer oHo3 (Part 2)
W3. Generalization to PSO, SHADE and MetaBBO hosts
PSO and SHADE are now tested, together with MadDE, DE and jDE21:
| JANUS adapter | vs native host, 100D | vs native host, 500D | vs matched random infill, 100D | vs matched random infill, 500D |
|---|---|---|---|---|
| SHADE (scalar model) | 9/0 | 12/2 | 9/0 | 12/0 |
| PSO (scalar model) | 11/0 | 13/0 | 5/0 | 9/1 |
16 functions, 30 seeds, 20,000 FEs, paired Wilcoxon tests with Holm correction per contrast and dimension; the full host set is in Table B and manuscript Table 3, and adapter settings are in Table 35. The PSO configuration was selected on seeds 201–205 of the same 16 functions and evaluated on seeds 1–30, so its result concerns new seeds, not unseen functions. For PSO, infill replaces selected particle positions after the ordinary velocity and position update. For SHADE, infill is excluded from the native F/CR success-memory credit because those parameters did not generate it (App. A.3, App. G.3).
A MetaBBO method as the receiving host is not tested. Such an adapter must define how externally generated candidates enter the learned policy's state and credit assignment, which differs for each method. MetaBBO methods appear in the paper as external baselines (Tables 15 and 20), and integration into a learned optimizer is listed as future work in the conclusion.
Thanks for the author's response. I am sticking to my initial judgment and keeping the score unchanged.
Response to Reviewer oHo3
Thank you for taking the time to consider our revision. We appreciated your suggestion to evaluate JANUS with PSO, SHADE, and MetaBBO as hosts. It prompted us to add the PSO and SHADE experiments in Table 3, where we compare JANUS with each native host and with matched random infill.
If you have a moment, we would be grateful to know which concern you feel we have yet to address adequately. Even a brief comment would help us better understand your assessment and where the paper still needs work.
Response to Reviewer oHo3 (Part 1)
W1. Clarity, and what is new
We rewrote the abstract, introduction and method overview.
The novelty relative to prior model-assisted and learned optimizers is summarized in Table 1:
W2. Scenario-specific implementations versus the unified framework (MOO uses only exploration)
The reviewer is right that the MOO adapter uses only one of the two roles. The revision states why and no longer presents MOO as an instance of joint exploitation and exploration.
What is common to all scenarios is the pipeline: fit a local model to recent evaluations, turn the fitted geometry into proposals, and let the host evaluate and select. Which roles are active follows from the feedback (Sec. 3.2, Table 11):
A vector objective does not define a single descent target without a preference rule or scalarization, so the MOO adapters propose no exploitation step and leave trade-offs to the host's Pareto survival.
We also report what this exploration-only use achieves. On 18 low-objective and 21 many-objective tasks (5% infill, ten seeds, 20,000 FEs), fixed-reference hypervolume improves significantly over no infill on 1/18 and 2/21 tasks after Holm correction and is not distinguished from matched random infill (Sec. 5.4, Table 40). The same holds for metric-only infill on a DE host (Table B). Separately, in tuned 500D CMA configurations the fitted metric's orientation does not consistently outperform a rotated-metric control; that control changes both roles and so isolates neither (App. G.4). The paper's supported claims therefore concern the scalar and residual settings, where both roles are active; the conclusion states that metric-shaped exploration used alone is not shown to outperform random infill. The earlier MOO rank summaries are retained in App. C.4 and labeled as descriptive results of an earlier study with unadjusted tests.