- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates when low-dimensional probing benefits large-scale multi-objective optimization by separating three interacting factors: representation geometry, query allocation, and candidate retention. It develops analytical results for a shared-scalar probe, including a representation-induced error floor, sufficient conditions for single-block dominance, and a decomposition of retention losses. Controlled experiments across 64 benchmark configurations suggest that the value of additional probe queries is problem-dependent, while retaining already-evaluated candidates can provide more consistent improvements than the tested adaptive query-placement strategy. The study is carefully executed and offers useful diagnostic insights. However, its theoretical results are relatively elementary and restricted to a specialized probe construction. Moreover, the empirical findings are largely tied to a single optimization backbone and short evaluation budgets, limiting their broader significance.
Critical strengths
- S1. Well-motivated decomposition of low-dimensional probing mechanisms. The paper clearly distinguishes representation limitations, the opportunity cost of additional queries, and losses caused by withholding or selecting evaluated candidates. This decomposition provides a useful perspective for understanding why low-dimensional probing succeeds or fails, beyond conventional comparisons of terminal optimization performance (Sections 2–3).
- S2. Rigorous controlled experimental design. The matched-query comparisons between Scan and All-scan preserve the evaluated probe candidates, evaluation costs, and pre-injection states, enabling a meaningful investigation of candidate-offering effects. The inclusion of dominance-safe block selection, fixed-grid controls, and shared-normalization interventions further strengthens the experimental methodology (Sections 4.2–4.3).
- S3. Extensive empirical evaluation and statistical analysis. The main study considers 64 benchmark configurations with 20 paired runs per configuration. The authors report both retrospective archive and final-population metrics, employ multiple-comparison corrections, and provide additional sensitivity analyses. These efforts improve the credibility and interpretability of the empirical findings (Section 4 and Appendices D–F).
Critical weaknesses
- W1. Limited theoretical novelty and generality. The weighted-variance decomposition in Equation (2) follows directly from the standard bias-variance identity, while Proposition 1 relies on a common-minimizer condition that makes block dominance relatively immediate. Similarly, Proposition 2 largely follows from elementary Pareto dominance properties. Although these results are mathematically valid under their stated assumptions, they provide limited new theoretical insight beyond the particular shared-scalar construction. A stronger contribution would establish more general conditions or nontrivial performance guarantees for broader classes of low-dimensional probes.
- W2. Restricted empirical generalizability. The main experiments rely on a single optimization backbone, with a fixed dimensionality of , population size , and evaluation budget of 1000 function evaluations. Although 64 configurations are included, many are related benchmark variants or deterministic transformations rather than independent problem families. Consequently, it remains unclear whether the conclusions extend to different population-based optimizers, other dimensionalities, more diverse Pareto-front geometries, or real-world black-box optimization tasks.
- W3. The retention advantage is sensitive to the evaluation budget. The results in Appendix D.6 show that the empirical advantages of All-scan over Scan become substantially less consistent at larger budgets. In the five-case budget-sensitivity study, the Archive ND win/unresolved/loss counts change from 3/2/0 at 1000 evaluations to 0/5/0 at both 5000 and 10000 evaluations. This raises questions about whether candidate retention represents a persistent optimization advantage or primarily benefits the early stages of search under severely limited budgets. A broader budget-dependent analysis would substantially strengthen the conclusions.
- W4. The mechanisms underlying long-term retention benefits remain insufficiently identified. Although the matched-query design isolates the intervention on candidate offering, changing the offered pool also affects environmental selection, population composition, and downstream search trajectories. The shared-normalization and population-state controls address several confounding factors, but the detailed causal investigations are concentrated primarily on DTLZ2. The supplementary state interventions also leave most individual-factor contrasts statistically unresolved. Thus, the evidence supports a composite retention-policy effect more strongly than a general causal explanation of the mechanism responsible for terminal improvement.
- W5. Limited evidence of practical superiority over established optimization methods. The external comparisons in Appendix F.4 cover only five selected benchmark configurations and three established optimizers. Moreover, some external methods have not completed their core optimization stages at the smallest evaluation budget, making direct comparisons difficult to interpret. While the authors appropriately acknowledge these limitations, stronger evidence is needed to establish the practical significance of the proposed diagnostic conclusions relative to existing large-scale multi-objective optimization approaches.
- W6. Presentation could be more concise and focused. The paper introduces numerous policies, metrics, intervention variants, and statistical comparison families, many of which require extensive appendix material to interpret. This makes the central scientific message somewhat difficult to distinguish from implementation-specific observations. A clearer synthesis of the main conclusions, their assumptions, and their practical implications would improve accessibility.
Questions for authors
Q1. Generalizability across optimization backbones. The empirical evaluation primarily relies on a single optimization backbone. To what extent do the observed advantages of retaining already-evaluated candidates depend on the specific environmental selection and population-update mechanisms? Would similar retention benefits be expected under alternative multi-objective evolutionary algorithms, such as NSGA-II or MOEA/D?
Q2. Budget sensitivity of retention benefits. The larger-budget experiments in Appendix D.6 show that the statistically significant advantages of All-scan over Scan become substantially less consistent at 5,000 and 10,000 function evaluations. How do the authors interpret this attenuation? Does it suggest that candidate retention is primarily beneficial during the early stages of optimization, rather than providing a persistent advantage throughout the search?
Q3. Novelty and practical implications of the theoretical analysis. The weighted-variance decomposition and dominance-based propositions appear to follow relatively directly from established mathematical properties. Beyond explaining the behavior of the specific shared-scalar probe, can the authors clarify what new theoretical insights these results provide? In particular, could the analysis lead to an actionable criterion for determining when additional queries or multiple-block retention should be preferred?
Q4. Causal interpretation of population-state effects. The DTLZ2 experiments demonstrate that identical nondominated sets and zero immediate retention gaps can coexist with different terminal outcomes. However, the contributions of population composition, dominated-solution quality, and ordering remain difficult to disentangle, despite the supplementary interventions. Can the authors provide further evidence identifying which aspects of population state are primarily responsible for the observed terminal improvements?
Q5. Sensitivity to probe representation and coordinate selection. The theoretical guarantees rely on assumptions about the scanned coordinates and common objective minimizers, whereas the actual optimizer uses a data-driven coordinate-screening procedure. How sensitive are the empirical conclusions to inaccuracies in the preserved-coordinate set and the choice of anchors? Can the authors more explicitly connect the screening-failure analyses to the conditions under which the proposed theoretical results remain informative in practice?
Q6. Fairness and scope of external-method comparisons. Appendix F.4 includes comparisons against established large-scale multi-objective optimization methods, but some external algorithms do not complete their core optimization stages within the smallest evaluation budget, and certain runs exceed the nominal budget. How should these comparisons be interpreted when assessing practical competitiveness? Could the authors clarify which conclusions remain supported when considering only budgets at which the compared algorithms have completed their principal optimization mechanisms?
Thank you for your review about the problems of this paper. We ran new experiments on NSGA-II and at larger scales (D=5000, M=5), and summarized our ablation and stage-completion data. The results show that (1) the benefit of retaining all evaluated candidates generalizes to NSGA-II and to D=5000, M=5; (2) candidate retention acts mainly as an early-stage boost under tight budgets; (3) the theoretical propositions provide simple checks for stopping extra queries and keeping non-dominated blocks; (4) the DTLZ2 performance gain comes from the combined population state rather than a single isolated factor; (5) screening errors degrade performance when extra distance variables are fixed or true position variables are missed; and (6) at 10000 FE where all external methods complete their core stages, WOF and LSMOF win on GLSMOP1 while our methods remain competitive on the other four problems.
W2 & Q1 Due to the character limit, please refer to the response to Reviewer 1.
W3 & Q2 Probing and candidate retention happen only once right after initialization (120 or 160 FE). At 1000 FE, only 7 to 8 evolutionary generations follow, so the initial retained candidates strongly affect the output. At 5000 and 10000 FE, standard evolutionary search runs for 47 to 98 generations and gradually dominates the trajectory, reducing the impact of that single early injection.
W1 & Q3 This paper aims to separate when probing is limited by subspace geometry versus candidate selection, which directly gives a practical criterion for both query allocation and multi-block retention. Because finer 1D sampling only reduces grid error until it hits the representation floor (Equation 2–3) and gives zero discovery gain once a common optimal step size is evaluated (Proposition 1), an algorithm can check whether coarse steps improve the score over the midpoint q=0.5. If the score gain is zero (359/360 GLSMOP runs in Appendix F.2), extra queries should be skipped to save the remaining FE budget for evolution.
W4 & Q4 All-scan has 43.45 dominated probe members per run with a mean distance g of 4.62, while Safe's dominated members all come from initialization with a mean g of 82.80 (Table 2 and Table 10). The table gives the Safe/All-scan mean IGD ratios at 1000 FE:
In Appendix E.5, replacing the whole dominated pool with the All-scan population improved both metrics significantly. Changing single factors one by one (reducing the distance g of dominated solutions, shifting coordinates, or changing row order) was not statistically significant. The improvement comes from the combined effect of dominated-solution quality, coordinate structure, and velocity state.
Q5 Theoretical results depend on the coordinates actually chosen by the screen. When the screen keeps extra distance coordinates (Appendix F.3), fixing 98 variables at anchor values increases the constant term Cam in Equation 2 and gave much worse IGD than using only the 2 position coordinates. Conversely, when the screen misses true position coordinates (Appendix D.4), overwriting missed positions with the shared scalar q creates conflicting optima across anchors, so the single-block condition in Proposition 1 no longer holds.
W5 & Q6 Archive ND uses the exact first B evaluations from each method, while Native ND uses the final population at termination. Because WOF and LSMOF need more than 1000 FE to finish their first weight-optimization stage, the 1000-FE results only describe their evaluated prefixes before that stage finishes. The table shows how many runs finished their first core stage by each budget, and the Archive ND results at 10000 FE across the five cases and two internal references.
When looking only at 10000 FE (where all external methods finished their core stages in all 100 runs), the 30 comparisons give 20 internal wins, 6 unresolved tests, and 4 external wins. All 4 external wins came from WOF and LSMOF on GLSMOP1, while our internal methods stayed competitive on the other four cases.
W6 We will rewrite the main text using standard MOO terms, explain Archive ND and Native ND in plain words when they first appear, and keep secondary ablation variants and detailed test families in the appendix.