- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how the amount of paired multimodal data affects model performance under fixed per-modality data budgets. Based on Partial Information Decomposition (PID), the paper proposes a four-channel scaling law that separately models redundant, modality-unique, and synergistic information. Experiments across three multimodal tasks show that paired data is essential for learning synergistic information, while unpaired data can improve other information components. The proposed scaling law also achieves better predictive performance than existing multimodal scaling laws.
Critical strengths
-
The paper addresses an important but relatively unexplored aspect of multimodal scaling laws by explicitly considering the pairing ratio under fixed data budgets. The experiments provide meaningful insights into how paired and unpaired data contribute differently depending on the information structure of each task.
-
The proposed 4 channel scaling law provides a reasonable framework for explaining multimodal learning behavior through PID. The experimental results are extensive and show consistent improvements over existing scaling laws, especially in predicting performance under different pairing ratios.
Critical weaknesses
-
Although PID provides a reasonable motivation for the proposed four-channel scaling law, the theoretical connection between information decomposition and the decomposition of training loss into four independent power-law terms remains unclear. The proposed formulation appears to be mainly empirically motivated rather than directly derived from PID.
-
It remains unclear whether the observed threshold is an inherent property of multimodal information learning or mainly a consequence of optimization dynamics. This distinction is important for understanding how generally the proposed scaling law can be applied.
-
The estimation of PID components in real-world datasets relies on several assumptions. Especially, chance-level unimodal performance does not necessarily imply the absence of unimodal information. This raises some concerns about how reliably the estimated information components reflect the actual information structure of each task.
-
The experimental evaluation is limited to three supervised multimodal classification tasks, with model size largely fixed in the main experiments. Although additional architecture experiments are provided, it remains unclear whether the proposed scaling law generalizes to larger multimodal models or more complex training settings, such as generative multimodal pretraining.
Questions for authors
-
What theoretical assumptions support the decomposition of multimodal training loss into four independent power-law terms based on PID? It seems to be clarified whether this formulation is theoretically justified or primarily an empirical modeling choice would help establish the contribution.
-
How should the observed synergy threshold be interpreted in relation to optimization dynamics? Clarifying whether the threshold reflects an inherent requirement for paired data or a model-dependent learning behavior would strengthen the main claim.
-
How reliable are the estimated PID components in real-world multimodal tasks? Since unimodal performance may not fully reflect the information available in each modality, clarification on the assumptions behind these estimates would help support the proposed interpretation.
2 comments
Brief Additional Comment and Current Decision
Dear Authors,
Thanks for the rebuttal. Most of my concerns are resolved. Since my previous concerns were pretty minor issues rather than critical flaws, I still think that this paper is at least a borderline accept. Thus, I currently maintain my score as borderline accept.
Best,
Reviewer otkk
Just one more quick additional comment: Next time, it might be very helpful to provide some changed or added numbers (e.g., performance results) in this responses for convenience!!
Thank you for the detailed review. We have uploaded a revised manuscript with color-coded changes which addresses your concerns and also adds two new sub sections: the first (Section 4.3) investigating what factors of training move synergy gates, and the second (Section 5.3) including practical tips on when to use which versions of our law. The latter agrees with your framing to distinguish the empirical contribution as inspired from PID formulation as opposed to formally identified by it.
We now address each of your points:
PID theoretical connection: We have reframed the PID framing as motivation and evidence that a four-channel law is likely a better fit, not as an actual derivation of our new law. Our empirical results validate this result and our assumption of additive independence appears to be a good model of this phenomenon but not necessarily the only correct approach. We temper our language in our original claim that our laws are more "theoretically grounded" than existing laws. The additive decomposition is our specific assumption and contribution rather than derived from the original PID framework, which we make explicit in the paper now.
Synergy gate properties: We have included additional analysis on the synergy gate in Section 4.3. We find that the synergy gate appears to be a property of the task, while the actual threshold can be determined by optimization dynamics. We found that pretraining unimodal encoders halves the synergy gate threshold but does not eliminate it. Additionally, training curriculum changes when loss occurs in relation to training step, but not in terms of paired datum exposure. This indicates that the synergy gate is not a property of training curriculum, but is affected by pretraining, and likely other optimization dynamics.
Concern of PID estimation: We agree our PID estimation approaches are model-dependent for VQA and NLVR2, and we have added a sentence on how our PID estimates are defined relative to its model class (similar to usable information from Xu et al., 2020). However, all of our PID estimations follow intuition: it is well-known that VQA as a dataset is dominated by text (Goyal et al., 2017), which our PID estimations agree with, and NLVR2's label set was specifically designed so that both modalities were necessary (Suhr et al., 2019). In our new section (Section 5.3) we find that ultimately, for practical settings, PID values are helpful as a prior for loss amplitudes in order to reduce variance, but that knowing their exact values are not necessary.
Generalizability and scaling: While our work focuses only on supervised classification tasks, we note that several works in scaling law literature also operate on classification tasks (Hestness et al., 2017; Rosenfeld et al., 2020), including some that exclusively do so (Sorscher et al., 2022; Bahri et al., 2024). Like these works, we believe that our findings will be useful beyond the classification setting. Additionally, the scale of our models is within an order of magnitude of several other existing scaling law papers (Hestness et al., 2017; Sorscher et al., 2022, Bahri et al., 2024). In Appendix C.2, we investigate varying Qwen encoder sizes and model architectures and find the same conclusions as the main paper.