- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes RIME, a cross-modal exchange mechanism where, instead of communicating complete source representations, two modalities reciprocally predict each other's representations and exchange only the prediction residuals ("innovations") — i.e., the information the receiver cannot yet explain. Innovations are re-estimated at every interaction block. Building on RIME, MolRIME performs innovation-driven exchange between 2D topology (GINE) and 3D geometry (TorchMD-ET) with symmetry-preserving updates (invariant coefficients modulating equivariant vector bases), combined with geometric denoising, topology reconstruction, reciprocal-prediction, and multi-conformer objectives. Pretrained on PCQM4Mv2, it is evaluated on MD22 (forces), QM9 (12 properties), MoleculeNet (8 datasets), plus a controlled frozen-encoder ablation against a "Direct Exchange" variant and an innovation-dynamics analysis.
Critical strengths
1,Shifting the question from how modalities interact to what should be exchanged is a genuinely useful reframing; the residual-as-innovation formulation is simple, intuitive, and connects to ideas from information decomposition and predictive coding. 2,The equivariant vector update (invariant coefficients × equivariant bases, centered positions, bias-free projection) is carefully constructed, with a complete O(3)-equivariance verification in Appendix A.3. 3,The Direct Exchange baseline differs only in the exchanged signal (same encoders, projections, gates, protocol, same labeled subsets, frozen encoders, 3 seeds) — this is the right way to isolate the contribution, and the reported gains (6.1–8.5% MAE reduction on 4/6 QM9 properties at 10% labels) are credible evidence for the mechanism.
Critical weaknesses
1,Best on only 3/7 MD22 systems (with large regressions elsewhere, e.g., Buckyball 0.214 vs LiTEN 0.114); best/tied on ~3/12 QM9 properties with small margins (2.4%/2.2%); best on only 3/8 MoleculeNet datasets, trailing M2UMol on 4. The core mechanism does not clearly translate into SOTA performance. 2,The "6.1–8.5% MAE reduction" in the abstract is measured against the authors' own ablation variant, not against external baselines; a reader could easily misread it as improvement over SOTA. 3,The predictor is trained with a cosine (direction-only) objective, but the residual Δ = Y − Ŷ contains magnitude mismatch, which is then exchanged as if it were "unexplained information". Magnitude error is not semantic novelty. 4,Residual magnitude partially increases at Block 3 while the story predicts innovations should shrink as information becomes predictable. The explanation offered is plausible but hand-wavy. 5,Residual exchange is close in spirit to modality factorization (Barezi & Fung, 2019, cited) and information-theoretic decomposition (Yang et al., 2026, cited); the delta over these works should be sharpened.
Questions for authors
see Weaknesses.
2 comments
Response part 2
-
We agree that the cosine-based objective encourages directional alignment without explicitly constraining feature magnitudes, so prediction residuals may contain magnitude discrepancies that do not necessarily represent novel semantic information. Our motivation for using this objective is to learn cross-modal representational correspondence without requiring exact reconstruction of source features, whose scales may differ across modality-specific encoders. Accordingly, RIME treats prediction residuals as receiver-conditioned discrepancy signals for cross-modal exchange, rather than as a strict measure of unexplained information.
We retain residuals at their original scale because normalizing each residual to a fixed norm could remove potentially useful magnitude variations while amplifying small residuals. Importantly, the residuals are processed through learnable projections and adaptive gates rather than directly added to the receiving representations, allowing the model to regulate their contributions during cross-modal interaction.
-
As clarified in our response to Comment 3, residual magnitude does not directly quantify the amount of complementary or semantically novel information. Instead, it reflects the discrepancy between a modality representation and its cross-modal prediction, which can also be influenced by differences in feature magnitudes. Therefore, the partial increase observed at Block 3 should not be interpreted as direct evidence of increased semantic novelty or reduced effectiveness of cross-modal exchange.
We would also like to clarify that RIME recomputes prediction residuals at each interaction block as both modality representations evolve. Consequently, the residuals at different blocks are calculated from different representations and predictions, rather than reflecting the progressive reduction of a fixed reconstruction error. In this setting, a monotonic decrease in residual magnitude is not necessarily expected. Moreover, the high cosine similarity at Block 3 can coexist with a relatively large residual because cosine similarity measures directional alignment without constraining feature magnitudes.
-
We agree that RIME shares the broader motivation of leveraging complementary information across modalities with the cited works. Building on this important research direction, RIME introduces a reciprocal prediction-residual exchange mechanism that dynamically determines cross-modal messages based on the current representations of both modalities. This provides a distinct approach to cross-modal interaction, complementing existing factorization- and decomposition-based methods.
Specifically, Barezi and Fung (2019) investigate modality-based factorization for multimodal fusion, whereas RIME constructs cross-modal exchange messages through reciprocal prediction residuals. Yang et al. (2026) explicitly decompose multimodal interactions into redundant, unique, and synergistic components using an information-theoretic framework. In comparison, RIME does not require an explicit decomposition of these information components. Instead, each modality predicts the representation of the other, and the resulting prediction residual is used to construct a receiver-conditioned exchange message. Moreover, the predictions and residuals are dynamically recomputed at each interaction block, allowing the exchanged messages to adapt as the modality representations evolve.
Beyond the general exchange mechanism, MolRIME further contributes a symmetry-preserving realization for molecular 2D–3D representation learning. Specifically, prediction residuals are constructed in invariant feature spaces, while invariant coefficients are used to update equivariant geometric features, maintaining geometric equivariance during cross-modal interaction. We have expanded the related-work discussion and clarified these methodological distinctions in the revised manuscript.
We sincerely thank the reviewer once again for the thoughtful and constructive feedback. The reviewer’s comments have been particularly valuable in helping us clarify the positioning of our work, deepen our explanation of the prediction-residual exchange mechanism, strengthen the experimental comparisons, and better articulate the distinctions between our approach and existing multimodal methods. We believe these revisions have substantially improved the clarity and rigor of our manuscript.
Response part 1
We sincerely thank the reviewer for their careful evaluation of our work, recognition of the relevance of our research direction, and constructive feedback. The reviewer's main concerns focus on the consistency of performance across benchmarks, the interpretation and effectiveness of prediction-residual exchange, and the distinction from existing multimodal approaches. We have carefully addressed these concerns by updating benchmark comparisons, clarifying the interpretation of prediction residuals, refining the presentation of our experimental findings, and strengthening the discussion of our methodological contributions. We provide detailed point-by-point responses below.
We appreciate the reviewer's concern regarding performance consistency across benchmarks. To provide a clearer and more rigorous comparison, we have updated the MD22 and MoleculeNet results by retraining representative baselines using exactly the same training, validation, and test samples as MolRIME.
On MD22, we retrained MACE and LiTEN and updated the MolRIME results. Under this evaluation setting, MolRIME achieves the lowest force MAE on 3 of the 7 MD22 systems.
Updated MD22 results. Force MAE (kcal mol⁻¹ Å⁻¹; lower is better). MACE and LiTEN were retrained using the same training, validation, and test trajectory frames as MolRIME. ViSNet results are taken from the original publication and are included as cross-study references rather than matched-split comparisons.
On MoleculeNet, we retrained MoleculeSDE (VE), MoleculeSTM, and M2UMol using exactly the same training, validation, and test samples as MolRIME. Under this controlled comparison, MolRIME achieves the highest mean ROC–AUC on 5 of the 8 datasets.
Updated MoleculeNet results. Test ROC–AUC (%, mean ± standard deviation over three seeds; higher is better). All listed baselines were retrained using exactly the same training, validation, and test scaffold splits as MolRIME.
The variation in performance may also reflect the task-dependent benefits of multimodal exchange. For MD22, molecular connectivity remains fixed within each system, while atomic forces depend strongly on the changing 3D configurations. Consequently, 2D topology may provide limited additional information for distinguishing trajectory frames, potentially reducing the benefit of cross-modal exchange on certain systems. For QM9, the prediction targets span spatial, electronic, vibrational, and thermodynamic properties, for which the relative contributions of topology and geometry may differ. These observations offer plausible explanations for the mixed results, although they do not establish a causal relationship.
More broadly, MD22, QM9, and MoleculeNet cover substantially different molecular prediction settings, including force prediction, quantum property estimation, and molecular property classification, where the relative importance of topology and geometry can vary. MolRIME provides a unified framework for integrating 2D molecular topology and 3D geometry, enabling its application across these diverse prediction tasks. We believe this flexibility highlights the broader applicability and potential of the proposed multimodal exchange mechanism.
To avoid potential misunderstanding, we have revised the final sentence of the abstract as follows: "Controlled ablations show that exchanging dynamically updated prediction residuals can improve downstream performance over full-representation exchange or fixed residuals, supporting the effectiveness of the proposed core exchange mechanism."