- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the impact of cross-modal update order on heterogeneous multimodal learning, and introduces HeteroPathBench to study path dependence under controlled training conditions. Experiments across seven sensor conditions and three backbone architectures demonstrate the existence of path-dependent performance differences. The paper further explores strategies to reduce path sensitivity and validates the phenomenon in native sequential multimodal training.
Critical strengths
- This paper investigates the impact of cross-modal update order on multimodal learning, addressing an aspect that has received limited attention in previous research.
- The experiments are well designed to isolate the effect of update order while keeping other training conditions consistent, that makes the comparisons more convincing.
Critical weaknesses
- The paper shows that some modality combinations are much more sensitive to update order than others, but does not explain why. Section 4 studies local updates and optimizer effects, but does not explain what causes this large difference between modality combinations.
- Appendix A.7 states that the authors recorded other metrics, but the main results only report path sensitivity using Recall@5. The authors should also report the other metrics for the same 420 matched path pairs to show whether the findings are consistent across different evaluation metrics.
- In section 6, the additional validation using the existing multimodal training algorithm MLA is limited to the CREMA-D dataset, providing insufficient evidence that path dependence generalizes to other sequential multimodal training methods.
Questions for authors
- What factors explain the differences in path sensitivity across different modality combinations?
- Do these metrics show similar differences between training paths to those reported with Recall@5?
4 comments
Dear Reviewer X8rZ,
Thank you again for your valuable comments on our submission. We would like to kindly confirm whether we have sufficiently addressed all of your concerns (or at least part of them).
Should there be any remaining questions or areas requiring further clarification, please do not hesitate to let us know.
If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.
We sincerely look forward to your feedback.
The author solved my problem, I maintain a positive rating. I believe the update order indeed has a significant impact on performance, and this is a good motivation.
Reply to Reviewer X8rZ — Appreciation for the Follow-up
We sincerely thank the reviewer for the encouraging feedback and continued support. We are glad that our additional analysis and experiments have addressed your concerns and helped clarify the significance of update order in multimodal learning. Your thoughtful suggestions have been valuable in strengthening both our mechanistic understanding and experimental validation. We will incorporate these findings and clarifications into the revised manuscript.
Response to Reviewer X8rZ — Mechanistic Insights and Additional Validation
We thank the reviewer for the careful reading and encouraging assessment. Your questions about condition-dependent sensitivity, additional retrieval metrics, and evaluation beyond CREMA-D are particularly helpful. They encouraged us to examine how update-order effects vary across training conditions and tasks, and what measurable factors may explain these differences. We have investigated all three questions further.
1. Understanding differences in path sensitivity across training conditions
Our submitted study established order dependence under matched initialization, directional data streams, and training budgets. Across 420 matched path pairs from seven sensor conditions and three backbone families, the mean absolute R@5 gap was 2.477 percentage points (pp), with substantial variation across conditions. All 252 real-network, same-state AB/BA operator probes showed nonzero differences; mean path sensitivity was 0.38 pp under momentum-free SGD and 2.96 pp under AdamW. These findings linked local update composition and training dynamics to final path differences.
Following your suggestion, we examined which measurable factors accompany these differences. We analyzed early directional gradients from the original 420 matched pairs. SEN12 clear and cloudy provide a useful comparison: both use 600 retrieval candidates, and their early gradient cosine similarities are nearly identical (−0.111 and −0.112). Yet their mean gradient-norm log ratios differ (0.771 versus 0.525), alongside different R@5 path gaps (2.018 versus 0.819 pp). This points to directional gradient strength, alongside gradient alignment, as a useful explanatory clue. Candidate-set size alone does not explain the variation: among conditions with 1,000 retrieval candidates, the mean R@5 gaps range from 0.364 pp on Sunboat RGB–sonar to 1.945 pp on LLVIP visible–infrared.
We also added matched two-step AB/BA interventions at 16 intermediate model/data/seed states, comparing the complete state with frozen BatchNorm buffers and reset Adam moments. Nonzero AB/BA parameter differences persisted in all 16 conditions under each setting. These controls identify a persistent local order effect, while the optimizer comparison shows how training dynamics influence its eventual scale.
In the revision, we will explain these relationships more carefully, connecting local update interactions, gradient characteristics, and accumulated trajectory differences to provide a more detailed account of why path sensitivity varies across training conditions.
2. Extending the retrieval evaluation to R@1 and R@10
At your suggestion, we evaluated R@1 and R@10 on the same 420 strictly matched path pairs from the submitted study, with the evaluation protocol unchanged. The mean absolute differences between paired endpoints are:
The order effect is therefore visible at all three retrieval cutoffs. These endpoint differences appear in both top-ranked retrieval and a wider retrieval window. We will present this matched multi-metric comparison alongside the original R@5 analysis so readers can better understand the effect across ranking thresholds.
3. Additional evidence in an image–text setting
We appreciate your suggestion to examine the phenomenon beyond CREMA-D. Our submitted MLA experiment established order effects within native audio–visual sequential training. To broaden the modality evidence, we conducted a new image–text sequential-update experiment on MVSA, using 21 matched random seeds and comparing text-first and image-first update orders. Paired runs shared initialization, stochastic streams, modality exposure, and optimization budget; we report final full-test metrics.
The differences appear in overall accuracy and both class-sensitive metrics. This extends the evidence of order-dependent endpoints to a text–image modality pairing. We will explain its matched protocol and distinguish it from the native MLA implementation in the revision.
We appreciate how these suggestions have helped us examine the findings from several additional perspectives. In the revised paper, we will integrate the gradient and state-intervention results into a more detailed mechanism discussion, report the additional retrieval metrics under the original matched protocol, and explain how the MVSA evidence extends our understanding of update-order dependence across multimodal learning settings.