- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
I was assigned the review of this paper one day before the deadline.
Multiobjective optimization is a key problem in many applications. This paper aims to construct a finite set of solutions and Pareto tradeoffs by proposing an cooperation mechanism. Experiments on synthetic benchmarks should demonstrate the usefullness of the approach.
Critical strengths
The approach with the normal tangent is clearly derived and potentially can even be called elegant. The paper is not too dense and easy to read.
Critical weaknesses
I am not sure how the theorem / propositions directly links to what is done in the experiments and they just to seem to be restatements e.g. Proposition 2.1? And Theorem 3.1 or Lemma C3 This seems to be straightforwad from the definitions or restatements and not necessairly what is actually needed to cover the experiments? What would actually be very strong and maybe be an independent publication would be better coverage, faster converge but the above seems to be fairly straightforward results and not necessairly be applicable to how the experiments are implemented.
Several Ablations of each model choice seem to be missing.
It is unclear how munch tuning was done for the number of hyperparameters a full ablation / sensitivity analysis and it seems that code is not to be released or at least nowhere mentioend?
Questions for authors
see above
4 comments
Response by the authors (2/3)
W2: Systematic Ablations Across Synthetic and Neural Model Choices
W2: Several Ablations of each model choice seem to be missing.
Thank you for raising this point. Holding budgets, initializations, and readouts fixed, we ablated every core design choice across our continuous and neural Pareto-set learning pipelines.
Synthetic pipeline and scaling ablations. Table 2a isolates five synthetic choices: (i) cooperative direction, comparing orthogonal projection against equal-norm unprojected (Matched), raw unprojected, and scalar-only (None) updates from random starts and shared frozen states at ; (ii) two-stage scheduling, comparing pre-freeze states at against frozen refinement at ; (iii) individual formation mechanisms, removing Gaussian repulsion, archive-gap guidance, preference adaptation, or coordinate proposals one by one; (iv) disabling coordinate proposals across all 17 transfer benchmarks, showing that orthogonal projection independently improves set quality whenever nonlinear linkage (LZ09) or density bias (DTLZ4) couples lateral spreading with descent, while remaining neutral on separable problems; and (v) problem scaling across and .
Neural optimizer, population, and backbone ablations. Table 2b isolates four neural choices: (i) updaters crossed with orientation (Adam, Adam–Armijo, Direct Armijo) on Adult and six-objective MSLR-WEB10K; (ii) single-batch monotonicity (Adam–Armijo minus Adam), confirming that batch non-increase gives no test gain whereas preserving scalar normal progress consistently improves held-out trade-offs; (iii) population size ; and (iv) model scale, widening MultiMNIST by 32.9x in parameters and evaluating six-model ResNet18 populations on NYUv2 and Cityscapes.
Table 2a. Synthetic ablations across orientation, stage schedule, components, and scale.
| Ablated Choice | Benchmark & Protocol | MOGF/D (Orthogonal) | Ablated / Control Variants |
|---|---|---|---|
Direction (full/K) | LZ09-F7 (12 seeds, IGD) | Full 0.0294; tail 0.0294 | Matched 0.0400/0.0354; None 0.0469/0.0412 |
| Suite direction | 17 tasks (12 seeds, no prop.) | 7/17 dual wins () | Wins on DTLZ4, F2–F5, F7, F9 (8 neutral) |
Stage (K -> T) | LZ09-F7 / F9 (12 seeds, IGD) | At T: 0.0251 / 0.0091 | At K: 0.1630 (-84.6%) / 0.1693 (-94.6%) |
| Components | ZDT3 / F7 (6 seeds, IGD) | Full 0.0097 / 0.0320 | No gap 0.0102/0.0336; fix w 0.0459; no prop. 0.1752 |
Scale (d, M) | d=10,30,100; M=2,3,5 | F7 0.025 / 0.101 / 0.269 | Matched 0.035 / 0.118 / 0.284; DTLZ2 to M=5 |
Table 2b. Neural ablations across optimizers, orientations, population size, and scale.
| Ablated Choice | Task & Metric | MOGF/D (Orthogonal) | Ablated / Baseline Variants |
|---|---|---|---|
| 3 Updaters dir. | Adult (12 seeds, Traj. HV) | 0.6744 / 0.6743 / 0.6748 | Matched 0.6733 / 0.6733 / 0.6738 () |
6-obj dir. (Adam/Armijo) | MSLR (12 seeds, HV) | Val-Sel 8.606 / 8.534 | Matched 8.323/8.155 (); None 8.194/8.169 |
| Batch monotonicity | Adam–Armijo minus Adam | Adult +0.24/-0.09/-0.06 () | MSLR -0.07/+0.01/-0.03 (); all |
Population size N | MSLR (N=6, 9, 12, 18) | HV 5.73->8.15->8.81->9.60 | Monotonic 6D test HV () gain (500 ep) |
| Model scale | MultiMNIST, NYUv2, Cityscapes | CNN 7.780; 0.198 / 0.546 | Matched 7.773; PMTL 0.180/0.537; WFR 0.183/0.542 |
Response by the authors (3/3)
W3: Hyperparameter Tuning, Sensitivity, Zero-Retuning Transfer, and Code Release
W3: It is unclear how munch tuning was done for the number of hyperparameters a full ablation / sensitivity analysis and it seems that code is not to be released or at least nowhere mentioend?
Thank you for raising these reproducibility points. Below we detail our tuning scope, sensitivity sweeps, zero-retuning transfer across 17 benchmarks, and code release.
Tuning protocol and validation-only selection. Formation parameters were chosen once via staged Bayesian optimization (Optuna) on development runs—locking prior fields at each stage—and held fixed thereafter (Table 3b). On all neural tasks, screening, selection, and checkpoint readout maximized validation hypervolume on optimization seeds disjoint from evaluation, with the test split disabled throughout tuning. Early synthetic searches overlapped part of the initial six-seed cohort; the revised paper discloses this explicitly and re-verifies all mechanism claims on fresh 12-seed cohorts.
Sensitivity grid and 17-benchmark zero-retuning transfer. Sweeping the cooperation multiplier eight-fold from shared frozen states on LZ09-F9 (Table 3a) shows that orthogonal projection monotonically improves both front proximity (IGD) and worst-case uncovered gap (Fill h) while keeping rejected steps below 0.5%, whereas unprojected cooperation works only when heavily damped before collapsing into uphill rejections. Learning-rate and cap controls on Adult and MSLR-WEB10K likewise show stable test gains without step-size fragility. Most decisively, freezing four family configurations and transferring them with zero per-problem retuning and zero coordinate proposals across 17 canonical benchmarks (2,040 runs) ranks MOGF/D #1 overall, ruling out per-task tuning (Table 3b).
Table 3a. Cooperation-strength sensitivity on LZ09-F9 (12 shared-K seeds, cap ).
| (Cap) | MOGF/D IGD | Unproj. IGD | Fill h (Orth / Unproj) | Zero (Orth / Unproj) |
|---|---|---|---|---|
| () | 0.0285 ± 0.0042 | 0.0285 ± 0.0042 | 0.114 / 0.114 | 0.03% / 0.03% |
| () | 0.0229 ± 0.0048 | 0.0133 ± 0.0032 | 0.104 / 0.070 | 0.15% / 0.25% |
| () | 0.0177 ± 0.0040 | 0.0154 ± 0.0062 | 0.086 / 0.073 | 0.15% / 0.05% |
| () | 0.0119 ± 0.0025 | 0.0965 ± 0.0287 | 0.062 / 0.201 | 0.17% / 90.25% |
| () | 0.0091 ± 0.0021 | 0.1000 ± 0.0301 | 0.040 / 0.203 | 0.47% / 91.79% |
Table 3b. Tuning protocol and 17-benchmark zero-retuning transfer (2,040 runs).
| Protocol / Suite | Tuning & Config Rule | Standing (8 Methods) | Sensitivity / Control |
|---|---|---|---|
Synthetic Optuna | Staged search (rate, gap, width) | Disjoint confirm seeds | Locked for 12-seed controls |
Neural Optuna | Val-HV screen + confirm seeds | Test disabled in tuning | Stable across LR & |
ZDT4,6 & DTLZ1–6 | Frozen ZDT1,2 & DTLZ7 configs | #1 4/8, Top-2 6/8 | Zero retuning; wins DTLZ4 |
LZ09-F1–F9 (9 tasks) | Frozen F9 config (no proposals) | #1 6/9, Top-2 9/9 | Wins 6/9 controls (F7) |
All 17 tasks | 4 frozen configs total | #1 10/17, Top-2 15/17 | AvgRank 1.76/8 (WFR 2.59) |
Code availability and open-source release. We apologize for omitting an explicit code-release notice initially. An anonymized reproducibility package (Supplementary Material zip on OpenReview) is already uploaded—containing executable synthetic and neural code for MOGF/D and baselines, fixed configurations, Optuna logs, per-seed outputs, and verification scripts alongside Algorithms 1–3 in the revised paper—and we commit to public open-source release upon publication (to be stated explicitly in the Reproducibility Statement).
We hope our response has addressed your concerns. If we have missed anything, please let us know.
Official Response by Authors
Dear Reviewer CNtg,
Thank you very much for taking the time to carefully review our work and provide your constructive comments. Your feedback has been invaluable in helping us improve our work.
In response to your comments, we have included the following content:
- We showed how Lemma C.3, Proposition 2.1, and Theorem 3.1 form a constructive chain governing our two-stage method and explaining its wider coverage (-65% to -80% gap on LZ09-F9) and faster convergence (8.5x faster than PMTL on LZ09-F7) (Section 3, Appendix D.2; Response (1/3)).
- We added systematic fixed-budget ablations isolating cooperative orientation (7/17 dual wins without coordinate proposals), two-stage freezing (K -> T), scaling in (d, M), neural optimizers, and larger backbones (Appendices F.8–F.16; Response (2/3)).
- We detailed the validation-only tuning protocol with fresh 12-seed verification, confirmed 17-benchmark zero-retuning transfer (avg. rank 1.76/8), and uploaded an anonymized reproducibility package with public code release committed (Appendices B.1, F.7; Response (3/3)).
- Besides, as suggested by Reviewers DotQ and qTsf, we proved finite-backtracking satisfaction of Eq. (16) (Appendix D.2), added Algorithms 1–3, and expanded theoretical comparisons against MOO-SVGD, PNG, and MGDA/D (Appendix A).
May we kindly ask if our response has addressed the issues you raised? If you still have any further questions or concerns, we would be more than happy to continue the discussion.
Thank you again for your time and effort.
Best regards,
Authors of Submission 41554
Response by the authors (1/3)
Thank you for noting that our normal–tangent derivation is "clearly derived and potentially can even be called elegant" and easy to read. Below we address
(1/3)W1,(2/3)W2(ablations), and(3/3)W3(tuning, sensitivity, and code).W1: Constructive Theory, Coverage, and Convergence Speed
Lemma C.3,Proposition 2.1, andTheorem 3.1are not restatements: they form a constructive chain governing our two-stage algorithm and explaining both wider coverage and faster convergence.Constructive chain and implementation.
Lemma C.3proves gi=∇ϕi(xi) is a strictly positive softmax mixture of objective gradients, so scalar stationarity (ni=0) gives exact Pareto criticality in O(md) closed form without per-particle QPs (MGDA,ParetoMTL). Naive repulsion violates box bounds and pushes converged points off the front;Proposition 2.1projects cooperation onto free coordinates LX(xi) with ε2-damping and cap ∥ti∥2≤ρ∥ni∥2, ensuring zero cooperative work (⟨gi,ni+ti⟩=−∥ni∥22) and equilibrium invariance (vi=0⟺ni=0) for all ρ≥0. Since curvature L perturbs tangency over finite steps,Theorem 3.1pairs alignment σ=(1+ρ2)−1/2 with Armijo bound κ=min(Mσamin,L2β(1−c)σ2)>0 to yield O(H−1) residual decay over H=T−K frozen steps and local linear contraction (1−η)k−K0 on PL basins. Practically, formation (k<K) adapts preferences and scales to allocate basins; freezing ϕˉi=ϕiK at k=K activatesProposition 2.1andTheorem 3.1under 17-trial Armijo on analytic tasks and minibatchAdamorAdam–Armijoon neural models.Better coverage. Without claiming a global nonconvex coverage theorem,
Proposition 2.1lifts the coverage–descent ceiling of unprojected updates (ni+ci). On curved or density-biased Pareto sets, repulsive and gap forces (Proposition B.2) carry uphill terms (⟨gi,ci⟩>0): weak weights (ρ≪1) leave wide front gaps, while strong weights (ρ≥1) flip updates uphill and stall line search. Enforcing ⟨gi,ti⟩=0 unlocks strong level-set transport (ρ≥1) while keeping subproblems synchronized across preference rays.Faster convergence. Conserving full normal dissipation −∥ni∥22 in O(md) closed form eliminates cooperative drag, uphill stalls, and QP overhead, accelerating steps, epochs, evaluations, and wall-clock time (Table 1).
Table 1. Link from normal–tangent geometry to coverage, speed, and updaters (
12seeds).F9,K=3501)h0.040; IGD0.0091h0.114(none) /0.203(unproj.)-65%to-80%gap;#114/21F9,K=3501)0.0287in99steps;1.84Mtot0.0285(1,500, none);2.13Munproj.15xfewer steps vs scalar-onlyMSLR, 10−4)8.07(ep60),8.26(90)8.01ep150(none);8.13unproj.2.5xfaster epochs; less stallF725.3savg;F7(18.8s)0.0294PMTL215.1s;WFR(18.8s)2.0388.5xfaster vsPMTL;0%uphillF7/F9atT:0.025/0.0091K:0.163/0.169;Armijo0%rise