acceptodds
Under review as a conference paper at ICLR 2027

Minimax Update Balancing for Unified Multimodal Pretraining

Abstract

Unified multimodal pretraining optimizes image understanding (I2T) and generation (T2I) through one shared backbone and one shared optimizer, and balanced task sampling does not balance the optimizer, since under SOAP the realized update serves the two directions unequally. The optimizer averages gradients over accumulation segments rather than balancing tasks directly. As a result, the averaged preconditioned update can be dominated by the segment with the larger norm under the preconditioner's metric. To address this imbalance, we introduce ML-FOP-SOAP, which operates within each gradient-accumulation window. It hierarchically combines consecutive segment averages in a bottom-up manner. At each fold, it computes the difference between the two segment averages, extracts the component orthogonal to the SOAP update, and adds this component back with a coefficient determined by minimax update balancing. This coefficient minimizes the worst-case local surrogate cost across the two segments, preventing either segment from dominating the update. The coefficient has a bounded closed-form solution. It is zero when the two segments are already balanced, equalizes their local surrogate costs when feasible, and otherwise falls back to the standard SOAP update. When the costs can be equalized, the correction follows the preconditioned multiple-gradient descent direction. Importantly, the preconditioner is estimated from the uncorrected mean gradient, keeping the balancing correction separate from preconditioner estimation. On Janus-400M, ML-FOP-SOAP achieves the lowest T2I loss among all tested optimizers and balancing methods while also reducing I2T loss relative to SOAP. It improves the entire I2T–T2I loss frontier across sampling ratios. Its aggregate loss improvement over SOAP is comparable to SOAP's improvement over AdamW, and it matches Muon's performance. Ablation studies show that each design choice matters. Using a fixed coefficient or Euclidean inner products performs worse than SOAP, while applying a single correction over the entire accumulation window captures only part of the improvement. Explicitly contrasting I2T and T2I at every fold degrades performance in both directions, supporting our segment-based approach. Across all three tested architectures, ML-FOP-SOAP achieves the highest mean proportion of updates that benefit both directions, indicating more balanced optimization. Its T2I improvement also increases when scaling from 400M to 1B parameters, where it achieves a 12% improvement in FID-1K.

Primary area:
optimization
Keywords:
Multi-objective optimization, Second-order optimization, Minimax optimization, Gradient accumulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VdExOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies optimization imbalance in unified multimodal pretraining, where image understanding (I2T) and image generation (T2I) share the same backbone. The authors argue that balanced task sampling does not necessarily lead to balanced optimizer updates, as the two tasks may benefit differently from the same update direction. To address this issue, the paper proposes ML-FOP-SOAP, which introduces a minimax gradient correction into SOAP. The method constructs an orthogonal residual from the difference between two gradients and determines its coefficient through a closed-form minimax solution. This correction is applied recursively to gradient accumulation segments through multi-level folding. Experiments on Janus-400M, Emu3-800M, and Janus-Pro-1B show consistent improvements over SOAP, particularly for T2I generation.

Critical strengths
  1. Meaningful motivation. The paper addresses a relevant problem in unified multimodal pretraining. The distinction between gradient statistical imbalance and actual update imbalance is interesting, and studying how a shared optimizer serves different tasks is a reasonable direction.

  2. Clear mathematical formulation. The method is supported by relatively complete mathematical derivations. The authors derive a closed-form minimax correction under SOAP's preconditioned geometry and establish its relationship with MGDA. The overall formulation is reasonably clear.

  3. Reasonably thorough experiments. The paper compares against several optimizers, including AdamW, Muon, SOAP, MGDA, and CAGrad. The ablations cover the main design choices, and the method consistently improves over SOAP across the tested architectures. The authors also report convergence curves and computational overhead, which help assess its practical value.

Critical weaknesses
  1. A gap between the motivation and the actual method. The paper is motivated by balancing I2T and T2I optimization, but the proposed method does not explicitly balance the gradients of these two tasks. Instead, it applies minimax corrections to gradient accumulation segments with randomly mixed task compositions. More importantly, Table S3 in the appendix shows that explicitly balancing I2T and T2I gradients actually leads to worse performance, while the proposed segment-based approach works better. This is an interesting observation, but the paper does not sufficiently explain why segment-level balancing helps or how it relates to the original task-balancing motivation.

  2. Limited analysis of gradient geometry in high-dimensional spaces. The reported cosine similarities between I2T and T2I gradients are close to zero, which is consistent with the observation that gradients in high-dimensional spaces are often nearly orthogonal. Although orthogonality alone does not invalidate the proposed correction, the correction coefficient can become very small when the preconditioned gradient norms are also similar. The paper does not provide enough analysis of the gradient geometry or the distribution of correction coefficients during actual training. As a result, it remains unclear whether the method primarily addresses systematic task imbalance or fluctuations between randomly sampled micro-batches.

  3. Limited advantages over strong optimizers and relatively small experimental scale. While ML-FOP-SOAP consistently improves over SOAP, it does not clearly outperform Muon in overall optimization performance. On Janus-400M, its aggregate validation loss is slightly higher than Muon's (3.625 vs. 3.622), while its I2T loss is substantially worse (0.672 vs. 0.568). The main advantage appears to be in T2I generation rather than overall optimization. Moreover, the largest evaluated model has only 1B parameters, leaving its effectiveness at larger pretraining scales uncertain.

Questions for authors
  1. Why does segment-level balancing work better than task-level balancing? Table S3 shows that explicitly balancing I2T and T2I gradients leads to worse results, while balancing randomly composed gradient accumulation segments improves performance. Could the authors explain the mechanism behind this difference? In particular, how sensitive is multi-level folding to the ordering of micro-batches, and could the improvements be related to stochastic gradient noise rather than task balancing itself?

  2. How does the correction behave in high-dimensional gradient spaces? Since the reported I2T/T2I gradient cosine similarities are close to zero, could the authors provide the distributions of gradient cosine similarities, preconditioned norm ratios, and correction coefficients for the actual folding operands? In particular, when the two gradients have similar preconditioned norms, does the correction effectively reduce to standard SOAP, and how frequently does this happen during training?

Reviewer_8xVHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies unified pretraining of multimodal models. Specifically, this is for models which share the same backbone and optimizer for image understanding (I2T) and image generation (T2I). The authors argue that equal task sampling does not result in the optimizer helping both tasks equally. The paper builds on top of SOAP, with their method titled ML-FOP-SOAP. Inside each gradient-accumulation window, it takes the difference between two consecutive segment averages, keeps the part orthogonal to SOAP, and adds it back with a coefficient from a closed-form minimax rule. This is done recursively over the window. It builds on SOAP and an earlier method called FOP. Main experimental results are on a 400B Janus model trained on 10B tokens of CC12M and CC3M image-text pairs. The dataset has annotations for both tasks. Experiments are done with three seeds. Compared with SOAP, validation loss improves by 3.6% for I2T and 1.4% for T2I. The aggregate loss matches Muon, at 5.8% lower throughput than SOAP. The authors also report single runs at 800M and 1B parameter scales across different model families.

Critical strengths
  1. The loss improvement over SOAP is clear and consistent, especially at 400M. It holds over multiple seeds and every optimizer has its own learning rate sweep. This is a nice experiment and result.
  2. Figure 2 is really convincing. At 400M, the authors are able to show that the whole I2T and T2I loss frontier moves, and no change of sampling ratio makes SOAP reach it.
  3. The results are reported honestly. The authors say that Muon is better on I2T loss, that CLIP and CIDEr do not separate the methods and that the PDR gain is within seed spread.
Critical weaknesses
  1. My main concern is that the evaluation does not show a better unified model. The main evidence is validation loss. A lower loss does not mean better accuracy. It may move the final metrics very little, depending on how saturated the model already is. The understanding (I2T) metrics are weak. Captions are scored only on the first sentence, with no reason given, so anything important in later sentences is ignored. CIDEr and BLEU-4 are not a good choice for modern captioning models. The MMKC-VS benchmark is very recent and only has 147 MCQ questions. All scores are between 0.23 and 0.33 (main-table best is 0.286), close to chance (0.25). At 800M and 1B, understanding is only measured by CIDEr-first and BLEU-first. The larger-model results lean towards generation.
  2. The gains look to be small and only on generation. Against SOAP, the clearest gain is on T2I loss. On I2T, Muon is better at 400M and at 1B. The aggregate loss is a plain average of the two losses, so it mostly relies on improvement in T2I. This is against the core motivation of the paper, which is to make better unified models.
  3. The main setting is pretty small and costs are unclear with scale. The 800M and 1B results are single runs. Figure 2 is shown only for 400M models. The cost in the abstract is the 400M number. Table S5 shows 9.2% lower throughput and 15% more memory than SOAP at 1B (12% and 24% against AdamW). On Emu3-800M, it is smaller, so it is unclear how cost varies with model and scale.
Questions for authors
  1. Why is only the first sentence in I2T scored? Why not look at a broad set of benchmarks for I2T like detailed captioning, visual question answering, any other captioning benchmark? This would substantially improve the takeaways and make the results much more convincing for practitioners.
  2. If possible, it would be great to have Figure 2 generated for 800M and 1B. Again, this would make the results much more convincing.
  3. If possible, can you add SOAP with momentum, selected hyperparameters for AdamW and Muon, a finer learning-rate grid chosen on a separate split, and the tuning budget used for ML-FOP-SOAP? What is the warm-start checkpoint?
  4. Does the gain survive I2T-only or T2I-only training? If yes, the balancing story is not the reason.
Reviewer_qjN7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that when a unified multimodal model trains I2T and T2I with a single optimizer, SOAP's averaged update favors the side with the larger -norm even under balanced sampling (update-service asymmetry). ML-FOP-SOAP adds, within the gradient-accumulation window, a residual correction orthogonal to the SOAP step to the averages of adjacent segments, with a bounded closed-form coefficient given by a minimax rule, applied recursively fold by fold. Experiments are mainly on a custom Janus-400M (3 seeds), where ML-FOP-SOAP has the lowest T2I validation loss (6.580 vs. 6.671 for SOAP), plus one run each on Emu3-800M and Janus-Pro-1B.

Critical strengths

The experimental protocol is fairly rigorous: the main experiment uses 3 seeds, each optimizer is tuned separately on the same learning-rate grid, three sampling ratios are swept, and throughput and memory overheads are reported.

The closed-form coefficient is bounded and interpretable, and the derivation in Appendix D is complete.

Critical weaknesses

The experiments do not substantiate the claimed task-balancing mechanism. The method balances randomly composed segments rather than I2T and T2I directly. In Table S3, directly contrasting the two tasks worsens both metrics relative to SOAP, while Table S7 shows the largest T2I gain where SOAP already has negligible service bias. These results leave the proposed explanation for the gains unsubstantiated. Learning-rate or update-norm-matched SOAP controls are needed to rule out a larger effective step size as an alternative explanation. This weakens the paper’s central empirical claim.  The contribution beyond preconditioned MGDA is not adequately isolated. When equalization is feasible, the proposed direction coincides with MGDA under the paper’s chosen metric; Figure S1 shows that the distinguishing clipping regime occurs in only 1 of 252 windows. Yet the SOAP-MGDA baseline changes the metric, step-length restoration, and clipping simultaneously. Without a controlled comparison using the same metric, segment operands, and multi-level folding, with and without step-length restoration, the experiments cannot establish whether the proposed minimax rule provides a meaningful advantage over MGDA. This leaves the method’s claimed novelty and benefit insufficiently supported.

Questions for authors

In Table S3 both directions become worse when folds are paired by task, and on Janus-1B the improvement is largest although SOAP is already essentially balanced. How is this consistent with the "balancing update service" explanation? By how much is the per-step update norm enlarged relative to SOAP on average? Which learning rate was selected for each method? Can SOAP approach ML-FOP-SOAP with a finer learning-rate grid or after matching the update norm? With the same segment operands and multi-level folding, what are the results of MGDA under the metric (with and without the step-length restoration)?

Comments on the paper

1 comment
Area_Chair_swqcOpenReview ↗

Discussion Reminder

Dear Reviewers,

I hope this message finds you well. As we are now in the discussion period, I would like to kindly remind you to leave comments for the authors through OpenReview if you have not already done so.

Thank you very much for your time and effort. Please let me know if you have any questions.

Best

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.