acceptodds
Under review as a conference paper at ICLR 2027

Paired Multimodal Scaling Laws

Abstract

Existing multimodal scaling laws fit multimodality terms empirically after testing and never vary how much data is multimodally paired at fixed data budgets. We investigate how, under the same total data per modality, changing the number of paired data affects loss curves in multimodal classification tasks. We train models in three different environments and run experiment sweeps varying data sizes and pairing budget. Pairing ratios have a dramatic impact on loss and this impact is directly tied to how much information synergy the task contains. Only paired data is able to reduce synergistic loss, while unpaired data can reduce redundant or unimodal information up until unimodal floors. Unlike traditional scaling laws where loss drops immediately in power law decay, synergy acquisition is gated, requiring a critical threshold of paired data before synergistic loss falls at all. We introduce a new family of multimodal scaling laws where total data-attributable loss is the sum of four individual power laws corresponding to the four different information channels of redundancy, a unique channel per modality, and synergy, and show how this law is both more theoretically sound and empirically valid across our experiments. This law predicts multimodal loss in our experiments more accurately than existing laws, with 3.2% error on fit tests versus 10.4% error for the best pairing extension of published laws.

Primary area:
learning theory
Keywords:
multimodality, scaling laws, machine learning theory, multimodal pairing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_otkkOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how the amount of paired multimodal data affects model performance under fixed per-modality data budgets. Based on Partial Information Decomposition (PID), the paper proposes a four-channel scaling law that separately models redundant, modality-unique, and synergistic information. Experiments across three multimodal tasks show that paired data is essential for learning synergistic information, while unpaired data can improve other information components. The proposed scaling law also achieves better predictive performance than existing multimodal scaling laws.

Critical strengths
  • The paper addresses an important but relatively unexplored aspect of multimodal scaling laws by explicitly considering the pairing ratio under fixed data budgets. The experiments provide meaningful insights into how paired and unpaired data contribute differently depending on the information structure of each task.

  • The proposed 4 channel scaling law provides a reasonable framework for explaining multimodal learning behavior through PID. The experimental results are extensive and show consistent improvements over existing scaling laws, especially in predicting performance under different pairing ratios.

Critical weaknesses
  • Although PID provides a reasonable motivation for the proposed four-channel scaling law, the theoretical connection between information decomposition and the decomposition of training loss into four independent power-law terms remains unclear. The proposed formulation appears to be mainly empirically motivated rather than directly derived from PID.

  • It remains unclear whether the observed threshold is an inherent property of multimodal information learning or mainly a consequence of optimization dynamics. This distinction is important for understanding how generally the proposed scaling law can be applied.

  • The estimation of PID components in real-world datasets relies on several assumptions. Especially, chance-level unimodal performance does not necessarily imply the absence of unimodal information. This raises some concerns about how reliably the estimated information components reflect the actual information structure of each task.

  • The experimental evaluation is limited to three supervised multimodal classification tasks, with model size largely fixed in the main experiments. Although additional architecture experiments are provided, it remains unclear whether the proposed scaling law generalizes to larger multimodal models or more complex training settings, such as generative multimodal pretraining.

Questions for authors
  • What theoretical assumptions support the decomposition of multimodal training loss into four independent power-law terms based on PID? It seems to be clarified whether this formulation is theoretically justified or primarily an empirical modeling choice would help establish the contribution.

  • How should the observed synergy threshold be interpreted in relation to optimization dynamics? Clarifying whether the threshold reflects an inherent requirement for paired data or a model-dependent learning behavior would strengthen the main claim.

  • How reliable are the estimated PID components in real-world multimodal tasks? Since unimodal performance may not fully reflect the information available in each modality, clarification on the assumptions behind these estimates would help support the proposed interpretation.

2 comments

Thank you for the detailed review. We have uploaded a revised manuscript with color-coded changes which addresses your concerns and also adds two new sub sections: the first (Section 4.3) investigating what factors of training move synergy gates, and the second (Section 5.3) including practical tips on when to use which versions of our law. The latter agrees with your framing to distinguish the empirical contribution as inspired from PID formulation as opposed to formally identified by it.

We now address each of your points:

PID theoretical connection: We have reframed the PID framing as motivation and evidence that a four-channel law is likely a better fit, not as an actual derivation of our new law. Our empirical results validate this result and our assumption of additive independence appears to be a good model of this phenomenon but not necessarily the only correct approach. We temper our language in our original claim that our laws are more "theoretically grounded" than existing laws. The additive decomposition is our specific assumption and contribution rather than derived from the original PID framework, which we make explicit in the paper now.

Synergy gate properties: We have included additional analysis on the synergy gate in Section 4.3. We find that the synergy gate appears to be a property of the task, while the actual threshold can be determined by optimization dynamics. We found that pretraining unimodal encoders halves the synergy gate threshold but does not eliminate it. Additionally, training curriculum changes when loss occurs in relation to training step, but not in terms of paired datum exposure. This indicates that the synergy gate is not a property of training curriculum, but is affected by pretraining, and likely other optimization dynamics.

Concern of PID estimation: We agree our PID estimation approaches are model-dependent for VQA and NLVR2, and we have added a sentence on how our PID estimates are defined relative to its model class (similar to usable information from Xu et al., 2020). However, all of our PID estimations follow intuition: it is well-known that VQA as a dataset is dominated by text (Goyal et al., 2017), which our PID estimations agree with, and NLVR2's label set was specifically designed so that both modalities were necessary (Suhr et al., 2019). In our new section (Section 5.3) we find that ultimately, for practical settings, PID values are helpful as a prior for loss amplitudes in order to reduce variance, but that knowing their exact values are not necessary.

Generalizability and scaling: While our work focuses only on supervised classification tasks, we note that several works in scaling law literature also operate on classification tasks (Hestness et al., 2017; Rosenfeld et al., 2020), including some that exclusively do so (Sorscher et al., 2022; Bahri et al., 2024). Like these works, we believe that our findings will be useful beyond the classification setting. Additionally, the scale of our models is within an order of magnitude of several other existing scaling law papers (Hestness et al., 2017; Sorscher et al., 2022, Bahri et al., 2024). In Appendix C.2, we investigate varying Qwen encoder sizes and model architectures and find the same conclusions as the main paper.

Reviewer_otkkOpenReview ↗

Brief Additional Comment and Current Decision

Dear Authors,

Thanks for the rebuttal. Most of my concerns are resolved. Since my previous concerns were pretty minor issues rather than critical flaws, I still think that this paper is at least a borderline accept. Thus, I currently maintain my score as borderline accept.

Best,

Reviewer otkk

Just one more quick additional comment: Next time, it might be very helpful to provide some changed or added numbers (e.g., performance results) in this responses for convenience!!

Reviewer_GMSJOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how the amount of paired data affects multimodal learning when the per-modality data budgets are held fixed. Experiments on a controlled Digits task, VQA, and NLVR2 investigate how paired and unpaired examples contribute to learning redundant, modality-specific, and synergistic information. The authors observe delayed acquisition of synergistic information and propose a four-channel scaling-law family with optional synergy gates and effective data counts. They evaluate these functions through data-scale extrapolation, full-pairing extrapolation, and held-out pairing-curve prediction. The findings offer guidance for allocating multimodal data collection budgets.

Critical strengths

An important and practically relevant research question. Varying the paired-example count while holding the per-modality data budgets fixed distinguishes the value of alignment from the benefit of simply collecting more data. This is consequential for multimodal dataset construction, where obtaining paired examples can be substantially more expensive than collecting unimodal data. The study offers useful guidance on when pairing matters and when additional unpaired data remains beneficial.

A systematic empirical investigation with a useful predictive framework. The Digits environment allows separate examination of redundant, modality-specific, and synergistic channels, while VQA and NLVR2 provide complementary real-world evaluations. The modality-presence controls help distinguish pairing effects from training–test input mismatch. The proposed functions also improve prediction. The released experiment records and analysis code make the study a useful resource for further investigation.

Critical weaknesses

The information-theoretic interpretation needs more careful qualification. The controlled Digits task provides a clear interpretation of the information channels, but their identification in real-world tasks is less direct. The VQA loss increase after question shuffling is model-dependent and is not shown to identify a specific PID synergy quantity. Similarly, near-chance unimodal predictions on NLVR2 do not alone establish that all reducible information is synergistic. The paper should distinguish the empirical usefulness of a PID-inspired parameterization from a formally identified information decomposition. These limitations constrain the theoretical interpretation without negating the central observations about pairing.

Minor writing and formatting issues:

  • Line 139: a period is missing after “(Becker et al., 2024)”.
  • Line 142: “the fourth digit (S) are” should use “is”.
  • Line 266: “NVLR2” should be “NLVR2”.
  • Line 357: the definition of g(x)g(x) should be: g(x)=(δ/(x+δ))βg(x)=(\delta/(x+\delta))^\beta.
  • Line 856: “asociated” should be “associated”.
  • Line 873: “backpropogated” should be “backpropagated”;
Questions for authors
  • What assumptions justify interpreting the VQA shuffling loss difference as PID synergy? If this is a model-dependent proxy, which conclusions rely on its numerical value rather than only on the observed pairing curves?

  • Please fix the writing issues mentioned in weaknesses.

2 comments

Thank you for the detailed review. We have uploaded a revised manuscript with color-coded changes which addresses your concerns and also adds two new sections: the first (Section 4.3) investigating what factors of training move synergy gates, and the second (Section 5.3) including a "practical tips" section on when to use which versions of our law. The latter agrees with your framing to distinguish the empirical contribution as inspired from PID formulation as opposed to formally identified by it.

Specifically, in the empirical results, we now show that PID values act as useful priors for loss amplitudes when few runs are available, but that free-fitting is more accurate once the number of training runs roughly doubles the number of free parameters. We temper our language in our original claim that our laws are more "theoretically grounded" as well.

As for VQA shuffling, shuffling the questions within the majority-answer class is specifically meant to maintain each modality's relationship with each label but break the specific pairing of a question and image to a specific label. The loss increase, therefore, corresponds to how much each individual pairing contributes. However, this approach is model-dependent, as you have mentioned, and we have added a sentence on how our PID estimates are defined relative to its model class (similar to usable information from Xu et al., 2020).

We have addressed all writing concerns and all fixed typos are also highlighted in red. Thank you for catching these.

Reviewer_GMSJOpenReview ↗

Comment for rebuttal

The authors have adequately addressed my concerns through additional analyses and clearer qualifications of their claims; I therefore maintain my original rating of 4.

Reviewer_ibHrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how paired data proportions affect multimodal scaling laws under fixed data budgets. Based on Partial Information Decomposition (PID), the authors propose a four-channel scaling law that separately models redundant, modality-unique, and synergistic information. Experiments on Digits, VQA-v2, and NLVR2 demonstrate improved loss prediction over existing scaling laws.

Critical strengths
  1. The paper investigates the overlooked role of paired data in multimodal scaling laws.

  2. The proposed four-channel scaling law provides an interpretable perspective based on information decomposition and achieves better predictive accuracy than existing scaling laws across multiple experimental settings.

Critical weaknesses
  1. The connection between PID and the four-channel scaling law lacks rigorous theoretical justification. The additive decomposition of losses is largely assumed rather than derived, and constraining the model with estimated PID values does not consistently improve prediction accuracy.

  2. The generalizability of the proposed scaling law remains uncertain, as the experiments primarily focus on supervised classification tasks.

Questions for authors
  1. In synergy-dominated tasks such as NLVR2, unpaired samples provide little predictive information. Increasing the pairing ratio therefore also reduces the proportion of potentially uninformative supervision. How can this effect be distinguished from the benefits of learning synergistic information?

  2. In VQA-v2, image-only samples are assigned answers sampled from questions associated with each image. Since the same image can correspond to different answers, how does this label ambiguity affect the measured contribution of unpaired image data?

3 comments

Thank you for the detailed review. We have uploaded a revised manuscript with color-coded changes which addresses your concerns and also adds two new sub sections: the first (Section 4.3) investigating what factors of training move synergy gates, and the second (Section 5.3) including practical tips on when to use which versions of our law. The latter agrees with your framing to distinguish the empirical contribution as inspired from PID formulation as opposed to formally identified by it.

We now address each point in more detail below:

PID theoretical connection: We have reframed the PID framing as motivation and evidence that a four-channel law is likely a better fit, not as an actual derivation of our new law. Our empirical results validate this result and our assumption of additive independence appears to be a good model of this phenomenon but not necessarily the only correct approach. We temper our language in our original claim that our laws are more "theoretically grounded" than existing laws. The additive decomposition is our specific assumption and contribution rather than derived from the original PID framework, which we make explicit in the paper now.

PID-fitting performs worse: In our new section, we show that PID values act as useful priors for loss amplitudes when few runs are available, but that free-fitting is more accurate once the number of training runs roughly doubles the number of free parameters. In our new section, we show that the advantage of PID loss is to aid in the variance-bias tradeoff. When few runs are available, PID pinning is helpful to avoid overfitting.

Generalizability of the law: While our work focuses only on supervised classification tasks, we note that several works in scaling law literature also operate on classification tasks (Hestness et al., 2017; Rosenfeld et al., 2020), including some that exclusively do so (Sorscher et al., 2022; Bahri et al., 2024). Like these works, we believe that our findings will be useful beyond the classification setting.

Do unhelpful, unpaired examples hurt?: This is a good question; we find that in general, extra unpaired rows appear to be neutral, as shown by Figure 4 where adding 16k unpaired images/texts to NVLR2 does not move loss. In our new section investigating curriculum learning, we find that frontloading or backloading pairs can affect synergy loss temporarily, but this effect vanishes with interleaved data pairing. We explicitly included a "modality presence" bit in our architectures so that this effect could be learned quickly by the model.

VQA image-only labels: Thank you for noticing this: yes, our approach means that we are adding inherent noise to the unpaired image task because there is no way to differentiate between different question options paired with the same image. We believe this would likely affect the contribution of an unpaired image datum slightly, but probably not significantly. The reduction in entropy from the entire 3,129-answer vocabulary to the approximately 5-to-1 question-to-image ratio still provides a useful training signal. We have noted this in our implementation details. If the reviewer would like for us to investigate this further, we would be happy to do so.

Thank you for the clarifications and revisions. The revisions have addressed some of my concerns. However, concerns about generalizability remain. while Figure 4 suggests that additional unpaired data does not hurt performance, it does not fully distinguish the benefits of synergy learning from those of improved supervision quality as the pairing ratio increases. So I maintain my original rating.

We have run an additional Digits experiment at D=32k with a three-channel model, where we remove the synergy digit and its loss. As the pairing ratio increases, synergy learning stays fixed (at 0) while supervision quality changes, as the redundancy channel still benefits from paired data. We compare loss of that model to the R+U1+U2 loss of the four-channel model (i.e. with synergy) and compare the losses:

pThree-channelFour-channelDifference
0.121.891 ± 0.0091.872 ± 0.0130.019
0.251.878 ± 0.0271.825 ± 0.0080.053
0.381.875 ± 0.0021.809 ± 0.0010.066
0.501.871 ± 0.0121.806 ± 0.0100.065
0.621.866 ± 0.0201.819 ± 0.0080.047
0.751.872 ± 0.0021.836 ± 0.0010.036
0.881.890 ± 0.0121.818 ± 0.0060.072
0.941.875 ± 0.0151.834 ± 0.0200.041
1.001.882 ± 0.0031.853 ± 0.0050.029

We find the four-channel model performs slightly better, likely because extra signal from the synergy digit also provides more supervision on pure digit prediction. However, we find no trend in p for this difference. We use this to conclude that the change in loss observed while varying p is attributable to learning synergy and not supervision quality.

Comments on the paper

1 comment
Area_Chair_prR1OpenReview ↗

Discussion Reminder

Dear Reviewers,

I hope this message finds you well. As we are now in the discussion period, I would like to kindly remind you to leave comments for the authors through OpenReview if you have not already done so.

Thank you very much for your time and effort. Please let me know if you have any questions.

Best

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.