- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies optimization imbalance in unified multimodal pretraining, where image understanding (I2T) and image generation (T2I) share the same backbone. The authors argue that balanced task sampling does not necessarily lead to balanced optimizer updates, as the two tasks may benefit differently from the same update direction. To address this issue, the paper proposes ML-FOP-SOAP, which introduces a minimax gradient correction into SOAP. The method constructs an orthogonal residual from the difference between two gradients and determines its coefficient through a closed-form minimax solution. This correction is applied recursively to gradient accumulation segments through multi-level folding. Experiments on Janus-400M, Emu3-800M, and Janus-Pro-1B show consistent improvements over SOAP, particularly for T2I generation.
Critical strengths
-
Meaningful motivation. The paper addresses a relevant problem in unified multimodal pretraining. The distinction between gradient statistical imbalance and actual update imbalance is interesting, and studying how a shared optimizer serves different tasks is a reasonable direction.
-
Clear mathematical formulation. The method is supported by relatively complete mathematical derivations. The authors derive a closed-form minimax correction under SOAP's preconditioned geometry and establish its relationship with MGDA. The overall formulation is reasonably clear.
-
Reasonably thorough experiments. The paper compares against several optimizers, including AdamW, Muon, SOAP, MGDA, and CAGrad. The ablations cover the main design choices, and the method consistently improves over SOAP across the tested architectures. The authors also report convergence curves and computational overhead, which help assess its practical value.
Critical weaknesses
-
A gap between the motivation and the actual method. The paper is motivated by balancing I2T and T2I optimization, but the proposed method does not explicitly balance the gradients of these two tasks. Instead, it applies minimax corrections to gradient accumulation segments with randomly mixed task compositions. More importantly, Table S3 in the appendix shows that explicitly balancing I2T and T2I gradients actually leads to worse performance, while the proposed segment-based approach works better. This is an interesting observation, but the paper does not sufficiently explain why segment-level balancing helps or how it relates to the original task-balancing motivation.
-
Limited analysis of gradient geometry in high-dimensional spaces. The reported cosine similarities between I2T and T2I gradients are close to zero, which is consistent with the observation that gradients in high-dimensional spaces are often nearly orthogonal. Although orthogonality alone does not invalidate the proposed correction, the correction coefficient can become very small when the preconditioned gradient norms are also similar. The paper does not provide enough analysis of the gradient geometry or the distribution of correction coefficients during actual training. As a result, it remains unclear whether the method primarily addresses systematic task imbalance or fluctuations between randomly sampled micro-batches.
-
Limited advantages over strong optimizers and relatively small experimental scale. While ML-FOP-SOAP consistently improves over SOAP, it does not clearly outperform Muon in overall optimization performance. On Janus-400M, its aggregate validation loss is slightly higher than Muon's (3.625 vs. 3.622), while its I2T loss is substantially worse (0.672 vs. 0.568). The main advantage appears to be in T2I generation rather than overall optimization. Moreover, the largest evaluated model has only 1B parameters, leaving its effectiveness at larger pretraining scales uncertain.
Questions for authors
-
Why does segment-level balancing work better than task-level balancing? Table S3 shows that explicitly balancing I2T and T2I gradients leads to worse results, while balancing randomly composed gradient accumulation segments improves performance. Could the authors explain the mechanism behind this difference? In particular, how sensitive is multi-level folding to the ordering of micro-batches, and could the improvements be related to stochastic gradient noise rather than task balancing itself?
-
How does the correction behave in high-dimensional gradient spaces? Since the reported I2T/T2I gradient cosine similarities are close to zero, could the authors provide the distributions of gradient cosine similarities, preconditioned norm ratios, and correction coefficients for the actual folding operands? In particular, when the two gradients have similar preconditioned norms, does the correction effectively reduce to standard SOAP, and how frequently does this happen during training?
Discussion Reminder
Dear Reviewers,
I hope this message finds you well. As we are now in the discussion period, I would like to kindly remind you to leave comments for the authors through OpenReview if you have not already done so.
Thank you very much for your time and effort. Please let me know if you have any questions.
Best