acceptodds
Under review as a conference paper at ICLR 2027

Dimension-wise Distributionally Robust Optimization for Multimodal Learning via Adaptive Aggregation

Abstract

Multimodal data inherently carries multiple semantic dimensions, such as imaging modality, anatomical region, and task type in medical data. Each dimension induces subpopulation shifts that may vary independently. Group Distributionally Robust Optimization (Group DRO) provides a principled framework for addressing such shifts by minimizing worst-case losses across predefined groups. However, when multimodal data carries multiple grouping dimensions, the standard practice of combining them via their Cartesian product leads to an explosive growth in the number of groups (for instance, combining 8 modalities, 5 anatomical regions, and 5 task types yields 200 groups), leaving each group with extremely sparse samples and making robust optimization unstable. To address this, we propose Dimension-wise DRO, which formulates robustness along each semantic dimension independently and adaptively aggregates dimension-wise risks into a unified objective. Our method consists of three components: (i) selecting non-redundant dimensions by pruning highly correlated axes; (ii) defining a smooth robust risk for each selected dimension via KL regularization; and (iii) aggregating dimension-wise risks with a Hedge-style adaptive weighting scheme that prioritizes harder dimensions. We theoretically prove that the proposed aggregated objective bounds the worst-case group risk across all dimensions, with a convergence rate of , matching standard SGD up to a logarithmic factor. Experiments on three multimodal benchmarks—OmniMedVQA, VQAv2, and BDD100K—demonstrate that our method consistently improves worst-group performance under both shifted and unshifted settings without sacrificing average accuracy, outperforming both annotation-dependent and annotation-free baselines.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Distributionally Robust Optimization, Multimodal Learning, Adaptive Aggregation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_f9FoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper addresses multi-dimensional robustness in multimodal learning. The classic baseline, Group DRO, forms the Cartesian product of all dimensions into a single grouping structure, but as the number of dimensions grows, this causes a combinatorial explosion with too few samples per group. The paper proposes Dimension-wise DRO: each dimension first performs a Group-DRO-like weighting on its own, reducing the losses of its groups to a single value; these per-dimension values are then aggregated into one total loss via Hedge-style adaptive weighting, with theoretical guarantees on the worst-case risk bound and convergence rate. Experiments on three multimodal benchmarks show consistent improvement in worst-group performance.

Critical strengths

The paper identifies the combinatorial group-explosion problem that arises when Group DRO is applied to multiple independent dimensions and proposes a dimension-wise formulation with a theoretical worst-case risk guarantee.

Critical weaknesses
  1. The paper's two-layer aggregation uses two different schemes: worst-case within each dimension, but a standard weighted sum across dimensions. The latter is, mathematically, the same problem as multi-task loss weighting (e.g., GradNorm, uncertainty weighting), yet the paper never discusses this connection or explains why the two layers use different aggregation schemes.
  2. Table 6 shows that a baseline trained on the best single dimension already reaches a worst-group accuracy of 24.9, while the full method reaches 28.4, a gap of only 3.5 pp, far smaller than this baseline's gap over the Cartesian-product baseline (24.9 vs. 14.1). The paper's core mechanism yields limited gains.
Questions for authors

None

1 comment

Thank you for the clarifications. The relation to multi-task loss weighting and the distinction between the inner and outer layers are now clear, and I have raised my presentation score. My main reservation remains, however: the improvement in Table 6 is modest relative to the extra complexity, so I will keep my overall score.

Reviewer_HYAbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

This paper proposes Dimension-wise DRO for improving robustness in multimodal learning. Instead of constructing groups through the Cartesian product of multiple dimensions, the method optimizes group robustness along each dimension separately and combines them using adaptive weighting. Experiments on three benchmarks show improvements in worst-group performance under both shifted and unshifted settings.

Critical strengths

Strengths

  1. I think the motivation is reasonable. The group explosion problem is relevant when multiple grouping dimensions are available, and optimizing each dimension separately is a practical direction.
  2. The experiments cover medical VQA, general VQA, and autonomous driving. The authors also provide ablation studies to examine different components of the method.
Critical weaknesses

Weaknesses

  1. Is this paradigm naturally generalize to the high-modality learning scenarios? It's a more practical and hard cases, where each time we have over two modalities [1,2], and some times they are missing [3]. It's good to compare these methods or discuss them in related work.
  2. More detailed analysis of worst-case performance and fairness would be helpful. For example, would it be possible to report the performance across different groups and their combinations? I wonder whether the proposed method consistently improves fairness across groups, or mainly benefits a few specific groups. Because groupDRO is used in this paper, and it is often leveraged to improve fairness and genneralization ability.
  3. I think more ablation studies on the grouping dimensions would be helpful. For example, could the authors compare using a single dimension and multiple dimensions under the same framework? I wonder how much of the improvement comes from using multiple dimensions rather than the proposed optimization strategy.
  4. I am also a little confused about the shifted evaluation setting. The authors remove the smallest group from training, but the proposed method relies on group losses to guide optimization. Can you elaborate more on the experiment settings as well as the takeaways?

[1] Liang P P, Lyu Y, Fan X, et al. High-modality multimodal transformer: Quantifying modality & interaction heterogeneity for high-modality representation learning[J]. arXiv preprint arXiv:2203.01311, 2022.

[2] Huang Y, Du C, Xue Z, et al. What makes multi-modal learning better than single (provably)[J]. Advances in Neural Information Processing Systems, 2021, 34: 10944-10956.

[3] Wu C, Shuai Z, Shen L. Rethinking Medical High-Modality Learning Under Missingness--A Long-Tailed Distribution Perspective[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026: 5641-5651.

Questions for authors

see weakness

Reviewer_2oLSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+2)
Read review
Paper summary

This paper proposes a novel Distributionally Robust Optimization (DRO) method for multimodal learning. As the authors describe, when considering each attribute of multimodal data as a dimension, traditional (group-)annotation-dependent methods suffer from the huge number of groups defined by the Cartesian product of all dimensions, while annotation-free methods may have difficulty in handling the balance/weights across different dimensions. To address these two well-introduced challenges, the authors propose a dimension-wise DRO framework: first, it identifies a subset of dimensions that are largely independent using a statistical metric, Cramér’s V. Then, it designs a soft worst-case group risk for each dimension, regularized by a KL-divergence penalty on the subgroup weights. Finally, the overall optimization objective is defined as a weighted sum of these dimensional risks, where the weights are updated using a Hedge-style rule. Experiments were conducted on three multimodal benchmarks, and the proposed method achieves promising performance compared to various baselines, including both annotation-dependent and annotation-free methods.

Critical strengths

(S1) Overall, this paper tackles important challenges in developing DRO to multimodal data and the proposed method looks sound. The three steps of the method are well designed and explained. Additionally, one important question I had during reading the introduction, whether optimizing the dimension-wise DRO is sufficient for optimizing the overall worst-case performance, is later answered by the proposed Theorem 3.1.

(S2) Overall, this paper is well-organized and easy to follow.

Critical weaknesses

(W1) The proof of the key proposed Theorem 3.1 looks incorrect, due to an inconsistency in the direction of the "less than or equal to" chain. Specifically, in Lemma B.14, let the LHS of Eqn. (1) be aa and the RHS be bb, then Eqn. (1) shows that a≤ba \le b. Later, the bb is derived into another inequality as in Eqn. (2). Again, let the RHS of Eqn. (2) be cc, then Eqn. (2) shows that b≥cb \ge c. However, how does the proof later come to that a≤ca \le c by combining (1) and (2) (line 1068)? Because both aa and cc are less than or equal to bb, how is the comparison between aa and cc determined? Do we consider any special equality case somewhere?

(W2) In Tables 1 and 2, only the average results from three runs (with different seeds) are reported. Without reporting variance and conducting statistical tests, it is difficult to evaluate whether the marginal improvements are statistically significant, e.g. 38.5 vs 37.9 for the Worst Acc of Unshifted VQAv2, and 0.211 vs 0.209 for the Worst mAP of Shifted BDD100K. As a result, these results, which only contain average values across different runs, cannot fully support the authors' claim that the proposed method achieves the best worst-group performance across all datasets and both settings (line 445).

(W3) One key result corresponding to the motivation raised in the introduction was not reported: what would Figure 1 look like after applying the proposed method? Does the proposed method address each dimension (and each subgroup inside) evenly? Considering that it should not be difficult to produce the figure given all the numbers already available from the experiments, I look forward to seeing an additional Figure-1-like plot for the proposed method, which would significantly strengthen the paper's comprehensive evaluation of the effects of the proposed dimension-wise DRO.

Minor weakness/comments:

(M1) in Figure 1, please provide the full names for those abbreviations, where there are plenty of spaces.

(M2) It would be better to introduce the definition of "dimension" (line 140) earlier in the introduction, to improve the smoothness of the reading.

(M3) In table 4, it should be better to replace the "Random seeds" to "# of random seeds".

Questions for authors

(Q1) For the choice of hyperparameters, I noticed that the weight for the KL divergence is very small (0.05 in Table 4). Why is that, and is 0.05 the rule-of-thumb value? Is the KL loss too large compared to the soft worst-case loss (Eq. 6)? Does the KL loss change much across different datasets when the number of dimensions changes?

1 comment

I appreciate the authors' efforts in the rebuttal, including the additional experimental metrics/results, and corrections/revisions to the theorem derivation and minor details. My concerns have been addressed, and I have increased my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.