- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper considers a variant of MGDA to address the stochastic multi-objective optimization (MOO) problem. It combines the variance reduction method SPIDER with a clipping technique to improve upon the sample complexity of existing methods. Also, the paper claims to achieve this better complexity result under a generalized smoothness condition, which relaxes the commonly used Lipschitz smoothness assumption. After the theoretical part, some numerical results are reported.
Critical strengths
The theoretical results are impressive: the paper not only achieves better sample complexity but also removes some assumptions.
Critical weaknesses
The paper is not well organized. Specifically, many parts are redundant (for example, the formulation in Line 202 repeats the equation in Line 36), the comparison with related work is relegated to Section 4.2 without any explanation, and the theoretical insights are not clearly explained.
The main result does not clearly resolve the -dependence. The claim in the FAQ is not supported and may be incorrect. Specifically, the appendix requires the initial value of to depend on , which, combined with the choice of , may introduce additional -dependent terms into the final results. Also, the exponent of (5/2, and may be even larger) is large, which limits the scalability of the algorithm.
The numerical experiments are limited to small datasets and unrealistic settings. An LLM-based experiment with more objectives can substantially strengthen the numerical evidence.
Questions for authors
In Line 49, the authors consider the CA, while the subsequent contexts ignore this issue. I wonder if this proposed method still has some guarantee on CA?
Minor mistake in Line 15: Term “MGDA” should not be used here.
2 comments
Thank you for your rebuttal. I'll maintain my rating.
We thank the reviewer for recognizing the theoretical contribution, and we will reorganize our paper.
Dependence on K. Our primary focus is the oracle-complexity dependence on ϵ for a fixed number of objectives K, motivated by applications with a small, fixed set of objectives. Our ϵ−3 optimality result concerns this regime, with constants depending on K, the confidence level, and problem parameters. We will clarify this scope and distinguish accuracy optimality from optimal joint dependence on K and ϵ.
Experimental scope. Beyond the synthetic and Multi-MNIST experiments, the appendix includes Office-31 ablations using a pretrained ResNet-18. These evaluate query efficiency and the contribution of double clipping. We acknowledge that scalability to many-objective LLM training remains empirically unverified.
CA guarantee. Most MGDA algorithms feature a CA direction, and the update directions in our algorithm also inherit this property.