acceptodds
Under review as a conference paper at ICLR 2027

Do Cross-Modal Updates Commute? Path Dependence in Heterogeneous Multimodal Learning

Abstract

Heterogeneous multimodal learning increasingly relies on aligning signals from sensors with substantially different statistical and semantic properties. Existing work has primarily focused on what should be aligned and how strongly alignment should be imposed, whereas the effect of update order is largely obscured by the standard joint-training formulation. This paper studies whether different compositions of the same directional updates lead to the same learned solution. To isolate this question, we introduce a controlled identification protocol that fixes direction-indexed data and augmentation exposure while varying composition order. Across seven heterogeneous sensor conditions and three backbone families, reordered updates produce broad but strongly condition-dependent endpoint differences, establishing path dependence across the evaluated training settings. Complementary diagnostics examine local update compositions and endpoint sensitivity across optimizer configurations. Same-state probes detect local differences, while optimizer comparisons reveal substantial variation in endpoint sensitivity. Building on this characterization, we investigate whether path sensitivity can be controlled. Interventions at the optimizer-state, path-composition, and local-operator levels identify effective regimes, and compute-neutral antithetic block pairing reduces aggregate sensitivity by 15.5% with a small average utility change. Finally, native sequential multimodal training reproduces order-dependent endpoints across 21 fresh seeds. Fixed-scale and randomized balanced schedules yield distinct sensitivity distributions, connecting the phenomenon to schedule design in an existing alternating multimodal algorithm. Together, these results establish update order as a consequential optimization variable in sequential multimodal learning and demonstrate that schedule design can reduce its endpoint effect in the evaluated regimes while retaining matched directional exposure and the original update budget.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multimodal learning, cross-modal alignment, path dependence, non-commutative updates, optimization, heterogeneous sensors

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_X8rZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the impact of cross-modal update order on heterogeneous multimodal learning, and introduces HeteroPathBench to study path dependence under controlled training conditions. Experiments across seven sensor conditions and three backbone architectures demonstrate the existence of path-dependent performance differences. The paper further explores strategies to reduce path sensitivity and validates the phenomenon in native sequential multimodal training.

Critical strengths
  1. This paper investigates the impact of cross-modal update order on multimodal learning, addressing an aspect that has received limited attention in previous research.
  2. The experiments are well designed to isolate the effect of update order while keeping other training conditions consistent, that makes the comparisons more convincing.
Critical weaknesses
  1. The paper shows that some modality combinations are much more sensitive to update order than others, but does not explain why. Section 4 studies local updates and optimizer effects, but does not explain what causes this large difference between modality combinations.
  2. Appendix A.7 states that the authors recorded other metrics, but the main results only report path sensitivity using Recall@5. The authors should also report the other metrics for the same 420 matched path pairs to show whether the findings are consistent across different evaluation metrics.
  3. In section 6, the additional validation using the existing multimodal training algorithm MLA is limited to the CREMA-D dataset, providing insufficient evidence that path dependence generalizes to other sequential multimodal training methods.
Questions for authors
  1. What factors explain the differences in path sensitivity across different modality combinations?
  2. Do these metrics show similar differences between training paths to those reported with Recall@5?
4 comments
AuthorsOpenReview ↗

Response to Reviewer X8rZ — Mechanistic Insights and Additional Validation

We thank the reviewer for the careful reading and encouraging assessment. Your questions about condition-dependent sensitivity, additional retrieval metrics, and evaluation beyond CREMA-D are particularly helpful. They encouraged us to examine how update-order effects vary across training conditions and tasks, and what measurable factors may explain these differences. We have investigated all three questions further.

1. Understanding differences in path sensitivity across training conditions

Our submitted study established order dependence under matched initialization, directional data streams, and training budgets. Across 420 matched path pairs from seven sensor conditions and three backbone families, the mean absolute R@5 gap was 2.477 percentage points (pp), with substantial variation across conditions. All 252 real-network, same-state AB/BA operator probes showed nonzero differences; mean path sensitivity was 0.38 pp under momentum-free SGD and 2.96 pp under AdamW. These findings linked local update composition and training dynamics to final path differences.

Following your suggestion, we examined which measurable factors accompany these differences. We analyzed early directional gradients from the original 420 matched pairs. SEN12 clear and cloudy provide a useful comparison: both use 600 retrieval candidates, and their early gradient cosine similarities are nearly identical (−0.111 and −0.112). Yet their mean gradient-norm log ratios differ (0.771 versus 0.525), alongside different R@5 path gaps (2.018 versus 0.819 pp). This points to directional gradient strength, alongside gradient alignment, as a useful explanatory clue. Candidate-set size alone does not explain the variation: among conditions with 1,000 retrieval candidates, the mean R@5 gaps range from 0.364 pp on Sunboat RGB–sonar to 1.945 pp on LLVIP visible–infrared.

We also added matched two-step AB/BA interventions at 16 intermediate model/data/seed states, comparing the complete state with frozen BatchNorm buffers and reset Adam moments. Nonzero AB/BA parameter differences persisted in all 16 conditions under each setting. These controls identify a persistent local order effect, while the optimizer comparison shows how training dynamics influence its eventual scale.

In the revision, we will explain these relationships more carefully, connecting local update interactions, gradient characteristics, and accumulated trajectory differences to provide a more detailed account of why path sensitivity varies across training conditions.

2. Extending the retrieval evaluation to R@1 and R@10

At your suggestion, we evaluated R@1 and R@10 on the same 420 strictly matched path pairs from the submitted study, with the evaluation protocol unchanged. The mean absolute differences between paired endpoints are:

Retrieval metricMean absolute path gap
R@11.499 pp
R@52.477 pp
R@102.698 pp

The order effect is therefore visible at all three retrieval cutoffs. These endpoint differences appear in both top-ranked retrieval and a wider retrieval window. We will present this matched multi-metric comparison alongside the original R@5 analysis so readers can better understand the effect across ranking thresholds.

3. Additional evidence in an image–text setting

We appreciate your suggestion to examine the phenomenon beyond CREMA-D. Our submitted MLA experiment established order effects within native audio–visual sequential training. To broaden the modality evidence, we conducted a new image–text sequential-update experiment on MVSA, using 21 matched random seeds and comparing text-first and image-first update orders. Paired runs shared initialization, stochastic streams, modality exposure, and optimization budget; we report final full-test metrics.

MVSA final-test metricMean absolute order gap
Accuracy0.541 pp
Macro-F10.815 pp
Balanced accuracy0.779 pp

The differences appear in overall accuracy and both class-sensitive metrics. This extends the evidence of order-dependent endpoints to a text–image modality pairing. We will explain its matched protocol and distinguish it from the native MLA implementation in the revision.

We appreciate how these suggestions have helped us examine the findings from several additional perspectives. In the revised paper, we will integrate the gradient and state-intervention results into a more detailed mechanism discussion, report the additional retrieval metrics under the original matched protocol, and explain how the MVSA evidence extends our understanding of update-order dependence across multimodal learning settings.

Dear Reviewer X8rZ,

Thank you again for your valuable comments on our submission. We would like to kindly confirm whether we have sufficiently addressed all of your concerns (or at least part of them).

Should there be any remaining questions or areas requiring further clarification, please do not hesitate to let us know.

If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

The author solved my problem, I maintain a positive rating. I believe the update order indeed has a significant impact on performance, and this is a good motivation.

AuthorsOpenReview ↗

Reply to Reviewer X8rZ — Appreciation for the Follow-up

We sincerely thank the reviewer for the encouraging feedback and continued support. We are glad that our additional analysis and experiments have addressed your concerns and helped clarify the significance of update order in multimodal learning. Your thoughtful suggestions have been valuable in strengthening both our mechanistic understanding and experimental validation. We will incorporate these findings and clarifications into the revised manuscript.

Reviewer_YBB8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the impact of cross-modal update order on heterogeneous multimodal learning. The authors introduce HeteroPathBench to isolate update-order effects under controlled training conditions. Experiments across seven sensor conditions and three backbones reveal substantial path dependence. The paper further analyzes optimizer effects and proposes antithetic scheduling strategies to reduce path sensitivity, with additional validation on sequential multimodal learning.

Critical strengths
  1. The paper investigates an interesting and relatively underexplored aspect of multimodal optimization.

  2. The extensive experiments across heterogeneous sensor conditions and backbone architectures provide substantial support for the observed path dependence.

Critical weaknesses
  1. The motivation and practical significance of studying cross-modal update order are not sufficiently established. Although the paper demonstrates path-dependent behavior in sequential multimodal optimization, it remains unclear why this phenomenon constitutes a distinctive challenge in heterogeneous multimodal learning or why reducing path sensitivity is important for improving actual learning performance.

  2. ABP also makes the compared schedules more similar, confounding the claimed benefit of antithetic structure. A schedule-distance-matched control is needed.

Questions for authors
  1. The main experiments use a fixed budget of 528 optimization steps. Could the authors clarify whether path sensitivity persists with longer training.

  2. The proposed directional updates are constructed by selectively routing cross-modal alignment gradients while keeping unimodal learning objectives active. If the unimodal objective is replaced with a joint multimodal classification loss, does the observed path dependency still persist?

3 comments
AuthorsOpenReview ↗

Part I — Motivation and Practical Significance

We thank the reviewer for the careful assessment. We appreciate your recognition of the research question and the extensive experiments across different sensor conditions and model architectures. Your comments also raise an important question about the motivation of this study: why should we care about update order in multimodal learning? We would like to clarify this point and will further develop it in the revised Introduction and Discussion.

Why update order matters. Multimodal models often involve encoders with different representations and learning dynamics. These encoders also interact through shared objectives. In sequential training, an update to one encoder changes the model state used by the next update. This means that two training paths can start from the same initialization, use the same directional data streams and number of updates, and consume the same computational budget, yet produce different results because the updates are executed in a different order.

We believe update order is an important training factor that deserves more attention. When training performance depends on the chosen schedule, researchers need to understand this dependence to interpret experimental results and make better training decisions. It also raises a practical question: can we reduce these differences and obtain more consistent results without increasing the training cost?

Our work investigates this problem systematically. We first designed HeteroPathBench to isolate the effect of update order while controlling other training factors. We then studied this effect across seven sensor conditions and three backbone families. These experiments helped us understand how path sensitivity changes with the data, model architecture, update schedules, composition depth, and optimizer states. We also conducted operator-level probes and state interventions to examine the mechanisms behind the observed differences. The results show that optimizer memory and normalization statistics cannot individually explain the phenomenon.

We also explored how to reduce path sensitivity in practice. After evaluating several approaches, we found that antithetic block permutation (ABP) offers a useful solution under certain training conditions. In our frozen study with 3,528 runs, ABP reduces average path sensitivity from 2.488 to 2.102 percentage points (15.5%). It also improves worst-path R@5 by 0.268 pp while maintaining comparable average performance (+0.076 pp). The method requires no additional training steps.

These results give the study a clear practical value. By reducing sensitivity to update order, we can make sequential multimodal training more consistent and reduce the risk of unfavorable training paths. Our experiments also show that reducing path sensitivity and improving average task performance are different goals. A good training strategy should consider both.

We see several opportunities for further research. For example, future methods could select update schedules according to the current training conditions, or develop stronger ways to control path sensitivity. It would also be interesting to examine whether similar effects appear in other training procedures, including those with less explicit update ordering. Our experiments provide a foundation for studying these questions, and we will discuss these directions in the revised paper.

In Part II, we will address the remaining experimental questions with our new results, including schedule-matched controls for ABP, longer training beyond 528 steps, and experiments using a fused multimodal classification objective with a joint-training reference.

AuthorsOpenReview ↗

Part II — Additional Experimental Evidence

Thank you for the thoughtful experimental suggestions. They have helped us further examine several important aspects of our study. Following your feedback, we conducted additional experiments on schedule matching, longer training, and multimodal classification objectives. We summarize the findings below.

1. Schedule-distance-matched controls for ABP

We appreciate your observation that changes in schedule distance may influence the measured benefit of ABP. To examine this factor more carefully, we added controlled comparisons with matched schedule geometry, allowing us to study different update arrangements under comparable geometric constraints.

The new experiments cover two settings:

ComparisonMatched cellsChange in path sensitivity
S6: ABP vs. geometry-matched control32−0.812 pp
S12: ABP+LAS vs. geometry-matched control32−2.036 pp

In both settings, the tested interventions achieve lower average path sensitivity under matched schedule geometry. We also examined 126 equal-distance schedule cells and observed differences across update arrangements sharing the same distance.

These findings suggest that the arrangement of updates can affect path sensitivity even under controlled schedule geometry. Your suggestion has helped us better distinguish the effects of schedule distance and update structure. We will include these comparisons and their detailed protocols in the revised appendix.

2. Path dependence under longer training

Thank you for raising the question about the original 528-step training budget. We extended the training protocol to 1,056 and 2,112 optimization steps, maintaining matched initialization, directional data streams, and update schedules.

Training stepsMatched pairsMean absolute R@5 gap
1,056632.465 pp
2,112632.927 pp

Using the same 0.01 pp tolerance, we observe nonzero differences in 61/63 pairs at 1,056 steps and 62/63 pairs at 2,112 steps.

The results indicate that path dependence persists when the training budget is extended to two and four times the original length. This provides additional evidence for the phenomenon over longer optimization trajectories. We will add these results to the revised manuscript.

3. Path dependence with a joint multimodal classification objective

Your suggestion to examine a joint classification loss is particularly helpful for understanding the phenomenon in a broader training setting. We therefore conducted a new controlled experiment on CREMA-D using audio and visual encoders with a shared fusion classification head, trained with fused cross-entropy loss.

We compared sequential AV and VA updates under matched initialization and directional data streams, and evaluated the final six-class classification accuracy.

SeedAV–VA accuracy gapABP-pair accuracy gap
7116.183 pp1.210 pp
7124.435 pp5.914 pp
7139.274 pp7.258 pp
Mean6.631 pp4.794 pp

The mean AV–VA accuracy gap reaches 6.631 pp under fused cross-entropy, indicating that update-order dependence remains observable when both modalities contribute to a shared classification objective. The ABP-controlled paths have a lower mean gap of 4.794 pp.

We also trained a conventional joint-update reference on the same classification task to compare the resulting task accuracy. These additional experiments help us better understand the behavior of sequential updates under different multimodal training objectives.

We appreciate these suggestions, which have helped us strengthen the experimental study. We will incorporate the new results, experimental details, and corresponding analysis into the revised manuscript and appendix.

Dear Reviewer YBB8,

Thank you again for your valuable comments on our submission. We would like to kindly confirm whether we have sufficiently addressed all of your concerns (or at least part of them).

Should there be any remaining questions or areas requiring further clarification, please do not hesitate to let us know.

If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

Reviewer_tUoBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how the order of modality updates in sequential multimodal training affects the final multimodal synergy result. The authors build an experimental environment and prove that different combinations of updates in the same direction can produce different optimization endpoints, and they reduce path sensitivity in some scenarios through reverse pairing scheduling.

Critical strengths

• The experiments are relatively rich, and a relatively complete exploration has been conducted. It has certain value as an empirical analysis. • The paper is easy to understand.

Critical weaknesses

• The non-commutativity of updates itself is not surprising, and a deeper, predictable mechanism explanation is still lacking. • No transferable principle or strategy for order selection has been formed; reducing path sensitivity has also not been translated into clear task performance gains. The practical value of this work is limited. • The efficiency and actual advantages (such as accuracy) relative to joint training have not been verified.

Questions for authors

Can the findings be used to improve modality learning sufficiency and balance?

5 comments
AuthorsOpenReview ↗

Part I — Mechanistic Insights and Further Exploration of Order Selection

Thank you for the thoughtful review and for recognizing the clarity and breadth of our experiments. We share your interest in understanding why update paths differ and how to choose a useful order. We find these questions particularly interesting. Our original study gave us a starting point, and there is still much to explore. Your comments prompted us to investigate both questions further. We have now completed new mechanism tests and explored several additional scheduling strategies.

1. From the original findings to a deeper mechanism analysis

In the submitted paper, we established two connected findings. First, update order has a measurable effect even when initialization, directional data streams, update counts, and compute are matched. Across 120 legal schedule classes, the average range of final R@5 reaches 6.34 percentage points (pp). Second, the size of this effect depends on how updates interact with the training state. All 252 same-state AB/BA operator probes showed nonzero differences; long-run path sensitivity was 2.96 pp with AdamW and 0.38 pp with momentum-free SGD. These findings motivated us to study the interaction between local update composition, optimizer dynamics, and data conditions.

Following your feedback, we examined these links more closely. We added two-step AB/BA interventions from identical intermediate states under 16 model/data/seed conditions. We repeated each comparison using the full state, frozen BatchNorm buffers, and reset Adam moments. Parameter differences remained nonzero in all 16 conditions in each setting. These tests help us identify the role of individual state variables in local update-order effects.

We also revisited early directional gradients in the original 420 matched pairs. For example, SEN12 clear and cloudy use the same 600 retrieval candidates and have almost identical gradient cosines (−0.111/−0.112), but different gradient-norm log ratios (0.771/0.525) and mean R@5 path gaps (2.018/0.819 pp). This gives us a concrete clue about why sensitivity varies between training conditions: the relative strength of the two directional gradients matters alongside their alignment.

In the revision, we will connect these findings into a more detailed explanation, from immediate AB/BA interactions and training-state effects to the differences that accumulate over a full training trajectory.

2. From an effective fixed schedule to further order-selection strategies

In the submitted paper, the schedule-class analysis showed that legal update orders could lead to substantially different endpoints. This suggested that order is a design choice we can control. We developed antithetic block permutation (ABP), which rearranges existing A/B blocks into paired, balanced sequences without adding training steps. Our frozen 3,528-run study found a 15.5% reduction in aggregate path sensitivity (2.488 → 2.102 pp). This was our first practical result: a simple fixed rule can reduce sensitivity to update order.

Your question encouraged us to explore how this initial method could be developed further. We first asked whether previous validation experience could guide the choice between ABP and a local adjacent-swap schedule (LAS), obtained by exchanging neighboring A/B blocks. For each dataset, backbone, and schedule family, we chose a strategy using earlier seeds' validation results and fixed it before testing new seeds.

Across 63 held-out-seed conditions:

ScheduleMean path sensitivity
Original strict schedules3.134 pp
Fixed ABP2.438 pp
Validation-guided choice2.266 pp

This provides an initial way to use information from past runs when selecting an update order.

We then examined whether the scheduling choice could change during training. In a separate 42-condition, two-stage study, we compared using ABP throughout with using ABP in the first stage and LAS in the second.

Two-stage schedulePath sensitivityMean R@5
ABP → ABP1.861 pp25.196
ABP → LAS1.766 pp25.431

The design is simple: the early and later stages can use different arrangements of the same directional updates. We also ran geometry-matched controls for the revised schedules. Across 32 cells in each setting, the S6 ABP and S12 ABP+LAS variants reduced path sensitivity by 0.812 and 2.036 pp, respectively, relative to their matched controls.

These new experiments extend the original ABP result in two directions: selecting an order using earlier validation evidence, and changing the order across training stages.

We will use the revised mechanism and scheduling sections to explain how the original findings led to these designs, what information each strategy uses, and what the new comparisons tell us about choosing update orders in multimodal training.

AuthorsOpenReview ↗

Part II — Practical Benefits and Modality Learning

Thank you for encouraging us to examine the practical implications of update-order dependence. Our submitted paper established the effect under matched training conditions and introduced antithetic block permutation (ABP) as a simple way to control it. Your comments prompted us to go further: can a deliberate choice of order and scheduling strategy improve task outcomes, and how does that choice compare with joint training? We added task-level comparisons, matched joint references, and supervised multimodal classification experiments.

1. From path control to better task outcomes

In our original frozen 3,528-run study, ABP reduced aggregate path sensitivity from 2.488 to 2.102 percentage points (pp), a 15.5% reduction. It also improved worst-path R@5 by 0.268 pp while maintaining comparable mean R@5. Here, the worst path is the lower-scoring endpoint of two matched update orders, so it measures the consequence of an unfavorable schedule choice.

Following your suggestion, we tested the performance effect more directly. In 32 new paired six-block comparisons, ABP improved mean R@5 by 0.895 pp and worst-path R@5 by 1.545 pp, while reducing path sensitivity by 1.300 pp relative to the original schedules. The improvement on the weaker path is particularly useful: choosing a better-organized schedule can improve the final prediction quality of otherwise unfavorable training orders. This connects our path-control result with a concrete downstream benefit.

2. Choosing between sequential and joint optimization

We added conventional full-joint updates as a reference for accuracy and training cost. Across 16 matched retrieval conditions covering four datasets, two backbones, and two seeds, both methods used 528 optimizer steps and recorded gradient evaluations, and each comparison ran on the same GPU. Six-block ABP took approximately 702 seconds per trajectory, compared with 844 seconds for full joint—about 16.8% less measured training time, with ABP faster in 15 of 16 conditions. ABP achieves its schedule changes by reordering existing directional blocks, without additional training steps.

Full joint obtained higher mean retrieval R@5 in this comparison (37.15% versus 31.86%), while ABP had the lower measured training cost. This makes the accuracy–time tradeoff explicit. The supervised classification study below provides another perspective: under a matched 10,398-step budget, ABP-controlled sequential training achieved higher mean fusion accuracy than joint updates. Together, the two task settings show why the training strategy and update order are useful choices to evaluate according to the task objective and computational budget.

3. Choosing an order also changes modality learning

Your question about modality sufficiency motivated a closer look at the representations learned under different schedules. In a new six-class CREMA-D experiment, we trained audio and visual encoders with a shared fusion head and cross-entropy loss. All methods used the same split, three seeds, and 10,398 steps. Besides fused classification accuracy, we tested the models after masking either audio or visual features. Masking audio leaves visual information available, and masking visual leaves audio information available.

Training methodFusion accuracyAudio maskedVisual masked
Sequential AV58.51%56.27%18.19%
Sequential VA61.02%40.86%41.04%
ABP-controlled VA63.35%41.53%40.46%
Joint update57.44%48.61%33.15%

The results support two observations. First, a more favorable update order and ABP scheduling can improve fusion accuracy in this evaluation. The three-seed mean is 58.51% for AV, 61.02% for VA, and 63.35% for ABP-controlled VA (+2.33 pp over VA and +5.91 pp over joint). Second, the masking results show that schedules change modality reliance. Sequential AV relies heavily on visual features, whereas VA and ABP-controlled VA retain useful predictive ability with either modality. ABP-controlled VA preserves this broadly balanced reliance while achieving the highest fused accuracy among the compared methods.

These experiments extend our original result from measuring path differences to making useful training choices. We will explain in the revision how selecting and controlling an update order affects downstream accuracy, how sequential and joint optimization compare in training cost, and how the resulting models use information from different modalities. We appreciate your suggestions, which helped motivate this closer investigation.

Reviewer_tUoBOpenReview ↗

It is more like a technical suggestion we can try, rather than a convinced theoretical conclusion.

Thanks for the response. However, my main concerns remain. The authors mainly perform some comparisons and tell us the order matters. A more important thing is what a order is the more efficient path and will achieve a better multi-modal fusion performance. This work fails to clearly make such a conclusion and verify its effectiveness and transferability across different fusion methods, modalities. It is more like a technical suggestion, rather than a very convinced theoretical conclusion. Given this, I will keep my rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Author Summary and Sincere Appreciation

As the discussion period comes to a close, we would like to summarize the scientific contribution of our work, the substantial improvements made during the review process, and our sincere appreciation to everyone who has helped us strengthen this research.

The central contribution of this paper is to establish update order as an independent and consequential dimension of heterogeneous multimodal learning. We investigate whether composing the same directional updates in different orders leads to equivalent learned models. Through HeteroPathBench, we isolate this question under matched data exposure, initialization, and computational budgets. Experiments spanning seven heterogeneous sensing conditions and three backbone families reveal substantial, condition-dependent differences in learning outcomes. Our analyses further connect these differences to local update interactions, gradient characteristics, optimizer states, and their accumulation throughout training. These findings provide a systematic basis for understanding why update composition matters in multimodal optimization.

We also demonstrate that update order offers a practical opportunity to improve training. In our frozen 3,528-run validation, antithetic block permutation (ABP) reduces aggregate path sensitivity by 15.5% without additional training steps. The review discussions motivated us to explore this direction further, including validation-guided schedule selection and changes in scheduling across training stages. These experiments provide additional evidence that carefully selected update orders can improve training outcomes and reduce sensitivity to unfavorable trajectories.

The review process has substantially strengthened both the scientific evidence and the presentation of our work. Following the reviewers' thoughtful suggestions, we extended the analysis to R@1 and R@10, examined gradient interactions and optimizer-state effects, introduced schedule-distance-matched controls, and verified that path dependence persists under longer training budgets of 1,056 and 2,112 steps. We also expanded the investigation beyond the original settings through text–image experiments on MVSA and supervised audio–visual classification on CREMA-D. Additional comparisons with joint training, modality-masking evaluations, and order-selection experiments further clarified the practical implications of our findings. These additions have helped us develop a more comprehensive mechanistic explanation, strengthen our experimental evaluation, and communicate the significance of update order more clearly.

We feel genuinely fortunate to have encountered such thoughtful, responsible, and dedicated reviewers. Each reviewer examined our work from a different perspective and raised questions that inspired meaningful further investigation. Their careful assessments, constructive suggestions, and encouragement have contributed greatly to the development of this research. We particularly appreciate the time and patience they devoted to engaging with our responses and helping us articulate the contribution more clearly. This has been a truly valuable and rewarding scientific exchange.

Finally, we extend our sincere gratitude to all three reviewers, our Area Chair, the Senior Program Chairs, and everyone involved in organizing and overseeing the review process. We deeply appreciate the considerable time, expertise, and care required to evaluate research rigorously and constructively, especially at a conference of this scale.

Thank you all again for your thoughtful contributions, encouragement, and invaluable support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.