- Rating
- 2
- Confidence
- 4
- Informativeness
- 2(−1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes COIN, a multimodal optimization framework that coordinates parameter updates according to their first-order effects on the fused prediction. Rather than relying on modality-specific gradient magnitudes or learning speeds, the authors introduce module response spaces to characterize attainable changes in the shared output. They identify an output-preserving equivalence relation between module responses and derive a minimum-energy allocation principle to eliminate redundant functional movement.
Building on this formulation, COIN jointly determines task-aligned responses and their allocation across modules through a regularized least-squares objective. The resulting solution incorporates both module-task alignment and cross-module interactions through a small Gram-matrix linear system. The paper further provides theoretical results concerning uniqueness, basis invariance, bounded functional motion, and the cost of ignoring cross-module interactions.
Experiments on AVE, AVSBench, VGGSound50, and FOOD101 demonstrate improvements over several multimodal learning baselines. Additional analyses investigate allocation efficiency, module cooperation, response-space geometry, and finite-step behavior.
Critical strengths
-
The paper studies multimodal optimization through the output changes induced by individual modules, rather than comparing their parameter-space gradients or learning speeds. This formulation highlights interactions that cannot be captured by module-wise statistics alone. This offers a useful perspective on cooperation between multimodal components.
-
The proposed objective admits a well-defined solution, and the paper establishes several relevant properties, including minimum-energy decomposition, basis invariance for fixed response spaces, bounded functional motion, and an exact characterization of the cost of independent allocation. The mathematical arguments appear carefully developed, with important assumptions and limitations explicitly discussed.
-
Experimentally, COIN achieves the highest reported mean fusion accuracy among the compared methods on all four benchmarks. The additional matched-progress comparisons, module removal experiments, and finite-step diagnostics provide further support for the proposed functional coordination mechanism.
Critical weaknesses
-
The theoretical components presented are all existing. Although their application to multimodal update coordination is meaningful, the paper could more clearly articulate which theoretical insights are fundamentally new beyond these established constructions.
-
The practical scalability of COIN remains insufficiently validated.
-
The paper does not report measured training-time or memory overhead relative to competing methods. The experiments also primarily involve conventional two-modality classification architectures, leaving the applicability to larger contemporary multimodal models uncertain.
-
There is no dedicated conclusion section, which is inconsistent with the standard structure of a full paper.
-
Some reported accuracy statistics appear inconsistent with the stated test-set sizes and three-seed evaluation protocol. For example, Appendix J.1 (Table 4) reports 402 test samples for AVE, while Table 1 reports an average accuracy of 69.87% for ARL over three seeds. Assuming standard sample-level Top-1 accuracy, the total number of correct predictions should be an integer. However, 402 × 3 × 0.6987 = 842.6322, which is not an integer. This discrepancy cannot be explained by rounding to two decimal places, as the nearest valid averages are 69.82% and 69.90%. The authors should clarify the evaluation protocol and provide per-seed results to verify the numerical consistency and reliability of the reported findings.
Questions for authors
Please see the critical weaknesses above. If the authors can convincingly address these concerns, I would be willing to reconsider and potentially raise my score. If the authors fail to adequately address Weakness 5, I will lower my rating.
5 comments
Follow up
Thank you for the response. However, I still have several significant concerns that have not been adequately addressed.
-
The incorrect loading of the AVE dataset affects not only ARL but also all other methods, including the results reported in this paper. Therefore, the corresponding experimental results need to be corrected. However, the authors have neither provided an explanation for this issue nor made the necessary corrections. As a result, I still have serious concerns about the validity and reliability of the reported experimental results.
-
Regarding the authors' response to W4: In my previous review, I explicitly pointed out that:
"There is no dedicated conclusion section, which is inconsistent with the standard structure of a full paper."
The authors also correctly summarized this concern in their current response as:
"W4: The paper has no dedicated conclusion section."
However, their actual response is difficult to understand. My interpretation is that the authors have provided the content they intend to include in a conclusion section. Specifically, there is no explicit indication, such as "This is our conclusion section," to clarify how this issue has been resolved. Instead, the response consists of a bolded summary statement followed by some additional content. I am therefore unsure whether the authors may have mistakenly provided an incorrect or misplaced response to this comment. Could the authors please clarify?
- Regarding the manuscript revisions: As of the time of writing this review, the changes described in the authors' response have not been reflected in the revised manuscript.
If I have misunderstood or overlooked anything in the authors' response or revisions, please feel free to point it out and provide further clarification.
Thank you for the follow-up. We have uploaded the revised manuscript, including the corrected results, additional experiments, and a dedicated conclusion.
AVE Evaluation and Baseline Reporting. We reproduced all baseline methods. Since some reproduced results were close to those reported in the original papers, we initially retained the published values for those methods. We have now updated the manuscript using our verified three-seed results, with all AVE evaluations standardized to the complete 402-example test set. For example, ARL has been corrected from 69.87 ± 0.44% (395 examples) to 69.90 ± 0.43% (402 examples). The updated results are reported in Tables 1 and 7. Our main findings remain unchanged.
Other revisions. Table 2 (p. 9) presents six complete-training controls, and Appendix J.8, Tables 11–13 (pp. 37–38) reports measured time and memory, sensitivity, and same-base masked-modality results. We have also added the requested Section 7, “Conclusion” (p. 10).
Theoretical contribution. COIN is not presented as a new projection or least-squares algorithm. Its contribution is the task-dependent joint allocation of module responses in fused-output space: Proposition 4.3 establishes compensating updates that module-local statistics cannot determine, and Theorem 4.4 quantifies the exact local cost of ignoring cross-module interactions. The complete-training results in Table 2 provide corresponding empirical evidence. We hope these concrete revisions address your concerns.
The AVE test loader previously loaded only 395 examples. We have fixed this issue, and the current evaluation uses all 402 test examples. For ARL, the three seeds correctly classified 275, 275, and 278 out of 395 examples, yielding 69.87 ± 0.44%.
Our contribution is the task-dependent coordination of module responses in the fused output space. We build on generalized inverses, orthogonal projection, and regularized least squares, as discussed in Appendix I.
Table 2 distinguishes modality-strength modulation, objective-gradient reconciliation, and separated learning. Section 2 identifies the remaining coordination issue: module-local magnitudes or learning speeds do not determine how updates to different blocks combine in the fused prediction. COIN instead allocates their attainable output responses jointly.
Three results distinguish this setting:
The new full-training controls connect this characterization to final performance: Full COIN achieves the highest mean accuracy among all six variants on all four datasets, including both single-direction controls. The margin over Independent is very small on VGGSound50 (61.52% versus 61.50%), while the gains are larger on the other three datasets.
The submission also evaluates recent external baselines, including MoNet (ICML 2026) and IBML (TPAMI 2026), alongside DGL, InfoReg, and MMPareto (Table 1). The submitted benchmark comparison reports higher fusion means for COIN than these external baselines. The new controlled comparisons address which part of the method contributes to that performance.
We measured optimizer time and peak allocated memory, including derivatives for correction. The isolated profile uses the same H100 and averages three profiling windows per method.
The overhead varies across tasks: FOOD101 peak allocated memory increases by 1.014 GiB. Independent and Scalar have similar profiled time to Full, as fresh gradients and Jacobian–vector products dominate the additional computation (Appendix H.4).
With two candidates per module, k≤2M; the five-module implementation has k≤10. Gram construction costs O(NCk2) and the solve costs O(k3), alongside response estimation and parameter-direction storage. The reported measurements cover two-modality classification architectures; larger-scale and three-modality evaluations remain for future work.
Useful module updates also need to cooperate through their effects on the shared prediction. COIN formalizes output-preserving reallocations, selects a minimum-energy joint response, and implements it through a small candidate-space solve (Sections 3–5). Fig. 1/Table 7 establish the local advantage over common rescaling, while the full-training ablations show the resulting accuracy gains across the evaluated datasets. The analysis characterizes local response spaces and the experiments test two-modality classification networks.