acceptodds
Under review as a conference paper at ICLR 2027

OFFICIAL NEURAL MOCO RANKINGS ARE EXTRACTOR-CONDITIONED: A FINITE-K PROTOCOL AUDIT

Abstract

Neural multi-objective combinatorial optimization is used for multi-attribute route planning and resource allocation. Published evaluations report a finite approximate Pareto set extracted by preference-based scalarization: one survivor per preference vector. That official archive is a legitimate score of the advertised preference-to-solution map. The same tables, however, report set-level hypervolume, so a printed ranking mixes per-preference quality with set attainment. We re-score frozen official decoded pools of five open-source constructors (WE-CA, PMOCO, CNH, GIMF-P, and NHDE-P) with a zero-search set-level extractor, Preference-Agnostic Selection (PAS): classical greedy hypervolume subset selection over the entire pool, with no extra neural pass and no local search. PAS strictly exceeds official scalarized extraction on of finite- settings. Mean-rank reversals occur on close pairs with official margins of –; published – gaps at larger need not reverse, because the PAS increment is often similar across a pair. We recommend reporting official and PAS hypervolume in parallel as two legitimate targets. Unmoved-then-Align (U2N) is a post-hoc local refinement on top of PAS. Official finite- rankings are therefore extractor-conditioned.

Primary area:
optimization
Keywords:
neural combinatorial optimization, multi-objective optimization, hypervolume, evaluation protocol, Pareto set extraction, preference-conditioned models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_71MVOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates whether official finite-K rankings of neural multi-objective combinatorial optimization (MOCO) solvers depend on how final solution sets are extracted from decoded candidate pools. It distinguishes the neural constructor from the extractor and compares official preference-based WS/TCH/PBI selection with Preference-Agnostic Selection (PAS), which applies classical greedy hypervolume (HV) subset selection to the same frozen pool without additional neural decoding or local search. The experiments identify PAS-induced mean-rank reversals for closely performing constructors in certain settings, while most model-pair mean rankings reported in Table 1 remain unchanged under PAS. The paper also introduces Unmoved-then-Align (U2N) as a separate local-inefficiency diagnostic, combining dominance-monotone refinement, preference-guided search restricted to T-unmodified solutions (W|R), and final HV-based selection. The authors recommend reporting official-extractor and PAS HV side by side to distinguish performance under the official preference-based extraction protocol from the HV of a set-level-selected K-archive on the same pool.

Critical strengths

1.It identifies a mismatch in existing reference-based neural multi-objective combinatorial optimization methods regarding solution selection and result evaluation (HV-based evaluation of solution sets). This finding is of great significance for comparing algorithms more objectively. 2. The paper evaluates five established neural constructors across multiple routing and knapsack problem settings. PAS improves HV in 53 of 56 finite-K settings, with three ties, providing evidence of extraction-related performance losses. The authors also report cases where model rankings remain unchanged and appropriately separate NHDE-P's unbounded external archive from fixed-K comparisons. 3. The analyses of PAS, refinement of the official archive, T-only refinement, residual preference-guided search, unrestricted W, and larger preference budgets provide useful insights into the relative contributions of candidate-pool selection and local refinement. These comparisons help distinguish pool-truncation losses from improvements obtained through additional local search.

Critical weaknesses

1.The writing is obscure and hard to follow, containing numerous disjointed and jumpy descriptions, which seriously hinders readers from grasping the intended meaning. Besides, many symbols are ambiguous or not defined in advance. 2.The paper demonstrates that changing the extraction protocol can reverse mean rankings, but the statistical evidence supporting a new superior constructor is less conclusive in some key cases. For example, Table 1 reports that the WE-CA versus GIMF-P mean HV difference on Bi-TSP20 changes from +0.00044 under official extraction to −0.00008 under PAS. However, Table A.39 shows that the 95% bootstrap confidence interval for the PAS difference is [−0.00017, +0.00002], which includes zero. The confidence interval for the change in the difference excludes zero, supporting an extractor effect, but not establishing a statistically reliable new winner. This distinction does not invalidate the observed mean-rank reversal. Nevertheless, the conclusions would be stronger if the paper more explicitly distinguished changes in observed rankings from statistically supported changes in relative superiority. The results also suggest that most larger-gap comparisons preserve their original mean rankings, so the practical scope of the ranking sensitivity should be emphasized. 2. Section 4.2 provides a reasonable motivation for restricting preference-guided search to solutions not improved by the dominance-monotone T operator. However, Table 3 and Table A.46 show that unrestricted Wall achieves higher HV in several settings. For example, on WE-CA Bi-TSP100, Wall obtains 0.70764 compared with 0.70678 for U2N, while both report 101 unique non-dominated solutions. This observation does not necessarily imply that Wall is preferable, since unrestricted search can reduce non-dominated solution counts in other settings and may require additional neighborhood evaluations. However, the current evidence does not fully isolate the benefit of the specific residual restriction. Reporting actual neighborhood evaluation counts and comparing the methods under similar search-effort budgets would provide a clearer justification for this design choice. 3. Although PAS requires no additional neural decoding or local search, it still incurs non-negligible subset-selection costs. Appendix E and Table A.15 report approximately 10–11 minutes of PAS postprocessing per 100 Bi-TSP100 instances, with full U2N requiring approximately 19–23 minutes. The paper appropriately distinguishes these offline CPU costs from the already-incurred GPU decoding budget and acknowledges that the current implementation is not fully optimized. Nevertheless, since the authors recommend PAS as a general reporting companion to official extraction, its practical computational overhead is relevant. The asymptotic complexity discussion does not fully establish the efficiency of an optimized implementation across different candidate-pool sizes and objective dimensions. Additional optimized runtime results or a more systematic cost–quality analysis would strengthen the practical recommendation.

Questions for authors

1.Given that the PAS mean-gap confidence interval for WE-CA versus GIMF-P on Bi-TSP20 includes zero (Table A.39), how should readers interpret the reported mean-rank reversal? Could the authors clarify how the statistical uncertainty affects the interpretation of the reported mean-rank reversals, as opposed to a descriptive change in mean ranking? 2.What is the actual neighborhood-search effort required by U2N compared with unrestricted Wall? Could the authors compare HV, unique non-dominated solution counts, and neighborhood evaluations under similar computational budgets? Such an analysis would help determine whether restricting W to the unmodified residual provides a favorable performance–cost trade-off. 3.Since the current PAS implementation incurs substantial offline processing time, what performance can be expected from an optimized implementation of greedy HV subset selection? Could the authors provide runtime measurements or complexity scaling with respect to candidate-pool size and objective dimensionality to support the proposed dual-track reporting protocol? 4.What does Aoff mean? The full names of PAS and U2N should be restated in the main text. What do M1–M3 stand for? What is N(x) in M2 and J_k(P) in M3? Symbols cannot be directly used without prior definition or citation. This makes the paper unintelligible. What is the intention of the paragraph “A K=2 count...” after Contributions?

Reviewer_9umUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the impact of archive extraction on the evaluation of neural multi-objective combinatorial optimization methods. It argues that existing preference-wise scalarization extractors may discard non-dominated solutions that contribute to set-level hypervolume. The authors evaluate Preference-Agnostic Selection (PAS), a greedy hypervolume subset selector, on frozen candidate pools from several neural MOCO solvers, and further introduce U2N for local refinement. Experimental results show consistent hypervolume improvements and occasional ranking changes under different extraction protocols.

Critical strengths
  1. The paper identifies a meaningful distinction between preference-wise solution quality and set-level Pareto approximation. The proposed dual-track evaluation, reporting both official and PAS hypervolume, could help disentangle these two objectives and improve the interpretation of future neural MOCO benchmarks.
  2. Experiments cover several representative neural MOCO methods and problem families. By reusing frozen decoded pools, this paper isolates the effect of archive extraction without introducing additional neural sampling / training.
Critical weaknesses
  1. PAS is essentially the classical greedy hypervolume subset selection applied to existing candidate pools, while U2N combines standard local search operators with preference-based refinement. The hypervolume gains of PAS are largely expected when selecting from a larger candidate pool.
  2. The main PAS-induced mean-rank reversal occurs on WE-CA versus GIMF-P for Bi-TSP20, where the new gap is only −0.00008 and its 95% confidence interval includes zero. Most other comparisons retain their original mean rankings, particularly at larger problem sizes. Thus, while the results show that extraction affects hypervolume scores, the evidence for substantially changing established solver rankings remains weak.
  3. Although PAS requires no additional neural decoding, its reported post-processing time is 10–11 minutes per 100 instances, compared with approximately 9 seconds for the official decoding pass. U2N incurs further overhead and is not consistently better than simpler alternatives, where applying local search directly to the official archive outperforms U2N on some CVRP and KP settings.
Questions for authors
  1. Could the authors show statistically significant PAS-induced ranking reversals across independent model checkpoints or additional benchmark settings, rather than primarily close or statistically inconclusive comparisons?
  2. Since that U2N might be inferior to local search applied directly to the official archive, what is the practical advantage of its more complex refinement and selection pipeline?
Reviewer_kW1SOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Audits finite-K evaluations of neural multi-objective combinatorial optimization. Official tables extract one WS/TCH/PBI survivor per preferece nvector, then score that K-archive by set-level hypervolume, conflating per-preference map fidelity with set attainment. The authors re-score frozen official decoded pools of five constructors (WE-CA, PMOCO, CNH, GIMF-P, NHDE-P) with Preference-Agnostic Selection (PAS), a zero-search greedy hypervolume subset selection, and with U2N, a post-hoc dominance-monotone local refinement plus residual alignment. PAS strictly exceeds the official extractor on 53 of 56 finite-K settings; close-pair rankings with official margins of 4–5×10^-4 reverse under PAS, while larger published gaps survive. Concludes official finite-K rankings are extractor-conditioned and recommends dual-track reporting.

Critical strengths
  • Uses official frozen checkpoints across five constructors and four problem families, so the numbers are reproducible rather than a synthetic re-implementation.
  • The three-mismatch decomposition (pool truncation, local inefficiency, training objective) is clean, and the proofs that dominance-monotone moves are hypervolume-monotone while unconstrained source-λ search is not correctly justify the U2N design.
  • The dual-track reporting template is a sensible convention the field could adopt.
Critical weaknesses
  • The headline 53/56 result follows from the comparison geometry rather than from the data: greedy hypervolume subset selection over a superset pool cannot lose to a scalarization-survivor archive, and the paper itself notes this as "expected subset-of-superset geometry". The central number is therefore a consequence of choosing PAS as the extractor, not an independent finding.
  • The methodological novelty is limited. PAS is greedy hypervolume subset selection (Guerreiro et al. 2016), U2N is Pareto local search with a residual restriction (Lust & Teghem 2010), and the fact that scalarization cannot recover non-convex Pareto fronts is standard in EMO (Das & Dennis 1997; Ishibuchi et al. 2018). The paper itself positions the contribution as quantifying the gap rather than introducing subset selection.
  • The ranking reversals are confined to close margins of 4–5×10^-4; Table 1 shows wider published 0.001–0.003 gaps survive and most pairs preserve their winner. The title's claim is therefore supported only for close comparisons, not for the rankings as a whole.
  • The reversal is not shown robust to the reference point. Hypervolume is reference-dependent and the audit uses each source paper's own r*, so it is unresolved whether the 73/200 Bi-TSP20 flip is a property of the extractor or of the reference choice, and no sensitivity analysis is offered.
  • The training-side claim (M3) is supported by a single continued-L_WS run on one WE-CA basin (Appendix F.3), and the differentiable set-level surrogate that would substantiate it is deferred, so the training-objective divergence is not yet established.
  • KroAB is a single instance (n=1) yet contributes a "1/1 flip" without a statistical test, no multiple-comparison correction is reported across the 46 Wilcoxon tests, and m≥4 is left open, so generalization to many-objective settings is not shown.
  • The recommendation is a reporting convention rather than a new method: PAS and U2N do not improve any constructor's generative capacity but only re-score a frozen pool, and the absolute gains around 10^-3 may not change which method a practitioner adopts.
Questions for authors
  1. The 53/56 result follows from greedy subset selection over a superset, which cannot lose to the scalarization survivor. What is the paper's distinct contribution beyond demonstrating that close rankings can flip?
  2. Would the 73/200 Bi-TSP20 flip persist under a single common reference point across constructors, or is it an artifact of using each source paper's own r*?
  3. M3 rests on one continued-L_WS run on a single WE-CA basin. What evidence shows this generalizes beyond that checkpoint?
  4. KroAB contributes a "1/1 flip" with no statistical test. Why is a single-instance result reported alongside the multi-instance Wilcoxon tests, and what claim does it support?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.