acceptodds
Under review as a conference paper at ICLR 2027

Alignment Strength Is Not Enough: Backward Influence Allocation in Heterogeneous Multimodal Learning

Abstract

Multimodal alignment is commonly controlled through a scalar strength, yet paired modalities need not contribute equally useful information. This paper studies a complementary control dimension: backward influence allocation. We introduce encoder-specific admission variables that determine how strongly a shared alignment gradient enters each encoder while leaving the forward alignment objective unchanged. We show that isotropic admission is equivalent to scalar tuning, whereas asymmetric admission accesses encoder updates that a single scalar weight cannot in general reproduce. Dense multi-seed comparisons reveal residual representation trade-offs under heterogeneous sensing, while matched sensing can be nearly scalar-sufficient, indicating that the value of allocation is condition dependent. Motivated by this observation, we develop a lightweight training-time policy that maps task–alignment gradient interaction to encoder-specific admission levels and learns from training-only progress, without preassigning a teacher modality or adding inference-time components. On ViT-B/16 across RGB–event and RGB–sonar retrieval, the learned controller attains the highest mean validation and ID R@5 among five specialized multimodal optimization baselines on all three datasets. Against validation-tuned scalar alignment, it achieves higher mean ID retrieval on all three datasets and higher mean validation retrieval on two. These results establish backward influence allocation as a distinct and learnable control mechanism for heterogeneous multimodal alignment.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Representation Learning, Cross-Modal Alignment, Contrastive Learning, Multimodal Optimization, Gradient Modulation, Self-Supervised Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WGwyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates backward influence allocation as an additional control dimension in heterogeneous multimodal alignment, beyond the conventional scalar alignment strength. The central idea is to introduce encoder-specific admission coefficients that independently regulate how the gradient of a shared alignment objective enters each modality encoder, while keeping the forward alignment objective unchanged.

Overall, the paper presents a conceptually interesting perspective on multimodal optimization.

Critical strengths
  1. A clear and potentially useful optimization perspective on multimodal alignment.
  2. The effort to investigate when and why backward allocation matters. This transparency strengthens the credibility of the empirical analysis.
  3. The narrative is clear and the language is well-articulated.
Critical weaknesses
  1. Insufficient evidence that the learned controller acquires a meaningful state-dependent allocation policy. The main empirical contribution is the proposed learned controller, but the current experiments do not convincingly isolate the benefit of policy learning from the benefit of introducing additional stochastic or time-varying alignment coefficients.

  2. Table 9 shows that a simple rule-based controller achieves competitive performance and outperforms the learned MLP in several settings. These observations leave open whether the REINFORCE-based learning procedure provides a substantial advantage over simpler allocation strategies.

  3. Most of the evaluated baselines were proposed between 2020 and 2024, with DGL (2025) being the most recent. The authors are encouraged to include representative state-of-the-art methods from 2026, particularly those addressing modality imbalance and gradient interaction, under comparable experimental settings.

  4. Limited statistical support and incomplete evidence of practical superiority. The current results establish competitiveness, but not reliable superiority over carefully tuned scalar alignment.

  5. Table 11 shows that at an alignment weight of 5, the MLP controller underperforms scalar alignment on all three ID benchmarks. Can the authors explain this behavior and clarify under which conditions the learned controller is expected to be advantageous?

  6. Figure 5 requires substantial improvement in presentation quality. In the rightmost panel (“Backward allocation”), the arrows associated with (gA) and (gB) appear to originate from disconnected points, with no clear connection to their corresponding gradient sources. Moreover, several gradient labels contain vertically stacked superscripts, making the mathematical notation difficult to parse.

Questions for authors

Please see the critical weaknesses above. If the authors can convincingly address these concerns, I would be willing to reconsider and potentially raise my score.

6 comments
AuthorsOpenReview ↗

Response to WGwy (Part I): State Dependence and Policy Selection

We thank the reviewer for recognizing the backward-allocation perspective. Our paper first identifies encoder-specific backward influence as a new control dimension, then proposes Rule-τ, Linear-K10, and MLP as alternative realizations. We selected MLP for the main study. These are our own proposed strategies, not external baselines. Your concerns motivated two targeted tests: whether MLP uses gradient state and whether its selection holds up after stronger tuning of the alternatives.

1. Does the MLP learn meaningful state-dependent allocation (W1)?

The submission shows asymmetric MLP actions, but action asymmetry alone cannot establish useful state dependence. Keeping the MLP, REINFORCE, and action grid fixed, we intervened on its four gradient-state features (two compatibilities, two log norm ratios). StateBlind zeros the inputs; StateShuffle breaks their temporal correspondence with training. Both preserve stochastic policy learning.

Across five matched held-out seeds (204–208), MLP exceeds StateBlind/StateShuffle on DSEC ID R@5 by +0.411/+0.720 percentage points. Against StateShuffle it wins in four seeds and ties in one (paired 95% CI [+0.039, +1.400] pp). Sunboat gaps are +0.131/0.000 pp: the benefit of state information is condition-dependent.

To control for action frequency, we ran 42 action-matched replays (seven seeds per dataset × three variants): original order, global permutation, or permutation within consecutive eight-action blocks. Each uses the identical multiset of joint (αA, αB) actions per seed. Ordered replay exactly reproduces the original validation and ID R@5 in all 14 cases.

ID R@5 (%)OrderedGlobal shuffleLocal block shuffle
DSEC4.6364.2063.944
Sunboat1.7341.3431.828

Global shuffling lowers mean ID R@5 by 0.430 pp (DSEC) and 0.392 pp (Sunboat) despite matched actions; local shuffling has mixed effects. State interventions test gradient-information use directly, while replay shows that action timing also matters. These controls go beyond the original admission-gap diagnostic.

2. Reassessing our proposed allocation strategies under stronger tuning (W2)

In the submitted work, Rule deterministically maps gradient statistics to admissions; Linear and MLP learn from those statistics using linear and nonlinear policies. All three are implementations we proposed for BIA. MLP was selected for full-scale evaluation before final test results. Table 9 showed that multiple implementations are viable, but at the shared λ=5, Rule sometimes exceeded MLP. We therefore retested their relative performance after stronger tuning.

We searched 15 (τ, λ) Rule configurations, nine Linear λ values, and MLP's existing nine-weight grid, selecting only on development Validation (seeds 201–203). Frozen choices were tested on seven independent, matched seeds (204–210), with the same ViT-B/16 initialization, splits, optimizer steps, and ID evaluator. Scalar is included as the conventional non-allocation reference.

Held-out ID R@5 (%)MLPTuned RuleTuned LinearTuned Scalar
DSEC4.6363.9124.2164.353
Sunboat1.7341.5291.2501.511

After stronger tuning, MLP has the highest held-out mean on both datasets. Its paired gains over Rule/Linear are +0.724/+0.420 pp (DSEC) and +0.205/+0.485 pp (Sunboat), winning in 6/7, 5/7 DSEC seeds and 4/7, 5/7 Sunboat seeds, respectively. Paired 95% CIs include zero; these are mean-level gains, not proven universal superiority.

Crucially, Rule’s occasional advantage in Table 9 does not persist in the validation-selected, held-out mean comparison on DSEC or Sunboat. This supports our choice of the MLP while keeping the broader contribution clear: BIA opens an optimization dimension with several viable realizations. We will explain the state tests and frozen selection protocol in the revision to distinguish the allocation principle from the merits of our chosen implementation.

AuthorsOpenReview ↗

Response to WGwy (Part II): Recent Methods, Statistical Reliability, and Alignment Sensitivity

We appreciate the reviewer's careful suggestions. They encouraged us to extend the comparisons, strengthen independent confirmation, and examine how alignment strength interacts with encoder-specific admission.

3. Extending the comparison to recent methods (W3)

The submitted Table 1 compares five multimodal optimizers, including OGM-GE and DGL. We added two recent approaches: Shared Space Alignment (SSA; ICML 2026), which studies shared cross-modal representations, and MASAM (ICLR 2026), which uses modality-aware sharpness-aware updates.

We adapted their mechanisms to the common ViT-B/16 dual-encoder retrieval setting (SSA-Ret and MASAM-Ret). Within each dataset, the methods share data manifests, seed-matched initialization, training-step budgets, and ID evaluation. Method-specific settings were selected using development validation.

ID R@5 (%)BIA-MLPSSA-RetMASAM-Ret
DSEC (7 seeds)4.6363.7864.049
Sunboat (7 seeds)1.7341.5851.156

BIA-MLP achieves the highest mean on both datasets. We will explain the adaptations and computational costs in the revision, making the connections between these optimization mechanisms and backward allocation clear.

4. Statistical reliability and structural value (W4)

The submitted SEN12 experiments examined whether encoder-specific backward influence reveals trade-offs beyond dense scalar tuning. Across five seeds, 44 scalar weights and 20 asymmetric allocations per seed produced 2.329% mean relative ID hypervolume expansion. Under matched Clear/Cloudy conditions, expansion was 0.083%/4.226%, revealing a stronger effect when sensing quality deteriorates.

We also strengthened the learned-controller evaluation. Hyperparameters were selected exclusively on development seeds 201–203, then frozen for seven independent, seed-matched confirmations (204–210). Both methods received their own validation-selected settings.

Held-out ID R@5 (%)BIA-MLPTuned ScalarRelative gain
DSEC4.6364.353+6.5%
Sunboat1.7341.511+14.8%

The expanded confirmation shows higher mean retrieval for BIA-MLP against a stronger, independently tuned Scalar. Together with the SEN12 frontier results, it clarifies both the additional optimization choices opened by BIA and the performance of our selected learned realization.

5. What the fixed-λ results reveal about policy selection (W5)

The reviewer's observation about Table 11 raises an interesting question: how does the overall alignment weight affect the value of a learned admission policy? For encoder m, the effective alignment influence is βm = λαm. The weight λ sets its scale, while the learned admissions distribute it between encoders. We therefore revisited the fixed-λ setting through more thorough validation-based selection.

DatasetSubmitted λ=5: MLP / ScalarSubmitted selected MLPRelative change to new held-out: MLP / Scalar
DSEC3.49 / 3.96λ=2.5: 4.60+32.8% / +9.9%
Sunboat1.23 / 1.46λ=10: 1.67+41.0% / +3.5%

All scores in the first two columns are ID R@5 (%). The last column compares each submitted fixed-λ=5 result with its new held-out mean in Section 4. It describes changes across two experimental protocols; the matched comparison is reported above.

In the new study, we searched nine λ values on development seeds 201–203, froze the chosen settings, and evaluated them on seeds 204–210. The tuned Scalar also improves over its original fixed-λ reference, while BIA-MLP shows a substantially larger relative increase. More importantly, BIA-MLP outperforms the separately tuned Scalar by 6.5% on DSEC and 14.8% on Sunboat within the same independent confirmation protocol. This provides a clearer account of how alignment strength and encoder-specific admission interact, and why policy selection matters.

6. Clarifying the gradient paths in Figure 5 (W6)

We thank the reviewer for this careful observation. It helps us improve the figure's precision and make the method easier to follow. We will redraw the rightmost panel with two alignment-gradient paths traced from the shared alignment loss to the corresponding encoders, label them directly with αA and αB, and show branch-specific task gradients separately. We will also correct the superscripts and harmonize notation. The caption will state the mechanism clearly: the forward alignment objective remains unchanged, while its backward influence is allocated independently to each encoder.

We will incorporate these results and improve the explanation of the connection between BIA's structural flexibility, learned policy selection, and sensitivity to alignment strength.

Reviewer_WGwyOpenReview ↗

Follow-up on the Rebuttal

I thank the authors for their detailed rebuttal. I would appreciate further clarification on three remaining points:

  1. With an ID gallery of 383 samples, random retrieval yields an expected R@5 of approximately 1.3%, whereas BIA-MLP achieves 1.734% and tuned Scalar achieves 1.511%. The absolute improvement is therefore only 0.223 pp. Could the authors elaborate on the practical significance of this gain and whether the results provide sufficiently strong evidence of meaningful representation learning on this dataset?

  2. The additional hyperparameter search convincingly demonstrates that selecting λ is important. However, this does not fully explain the original observation in Table 11: at the same λ = 5, MLP underperforms Scalar on all three ID benchmarks. Could the authors provide a explanation for this behavior?

  3. At the time of writing this follow-up, the PDF available to reviewers still appears unchanged, for example, Fig. 5.

If I have misunderstood anything, please feel free to correct me.

Reviewer_zEFGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces backward influence as a complementary control axis to scalar alignment strength in multimodal learning. Instead of changing the forward alignment objective, the method gates how strongly the alignment gradient is admitted into each encoder via encoder-specific admission variables implemented with stop-gradient mixing

Critical strengths
  • Introduces a clear and simple mechanism (encoder-specific admission via stop-gradient) that decouples “how much alignment” from “where it is absorbed,” while leaving the forward objective unchanged.
  • The core idea is well-motivated and illustrated; mathematical statements are concise, and the forward-identical, backward-gated construction is easy to implement and reason about.
Critical weaknesses
  • Limited novelty and theoretical contribution: Unequal or one-way cross-modal gradient propagation already appears in locked-tower learning, cross-modal distillation, anchor-modality methods, and modality-specific gradient modulation. Its theoretical results—that symmetric coefficients reduce to scalar weighting and asymmetric coefficients cannot be represented by one scalar—are mathematically correct but largely immediate.
  • Incomplete and potentially unfair empirical validation: The mechanism analysis uses SEN12 with ResNet-18, while the final controller is evaluated on different datasets using ViT-B/16, leaving the two parts insufficiently connected. Moreover, several classification-oriented baselines are substantially reinterpreted for retrieval, direct alignment baselines such as locked towers and fixed asymmetric allocation are missing, and the tuning budgets are unequal.
Questions for authors
  • One-way or asymmetric cross-modal gradient propagation already appears in LiT [1] or ImageBind [2]. Modality-dependent gradient modulation has also been studied by OGM-GE. How does the proposed method fundamentally differ from these approaches beyond providing a continuous parameterization restricted to the alignment loss?
  • The distinction between scalar and asymmetric weighting appears to be an elementary algebraic consequence: a single scalar can only produce equal coefficients for both modalities, whereas two independent coefficients can represent unequal weighting. It is unclear whether it warrants a separate theoretical section or constitutes a meaningful theoretical contribution..
  • The primary mechanism analysis uses fixed allocation on SEN12 with ResNet-18, whereas the learned controller is evaluated on DSEC, Eilat, and Sunboat with ViT-B/16. Although Appendix D reports additional structural sweeps, these experiments use different galleries and protocols. Could the authors validate dense scalar tuning, fixed asymmetric allocation, and the learned controller under the same ViT backbone, dataset, and training protocol used in the final evaluation?
  • Several baselines were originally designed for supervised multimodal classification or general multi-objective optimization, but are substantially reinterpreted for contrastive retrieval. In particular, OGM-GE originally estimates modality strength through prediction confidence, whereas the paper substitutes within-modality contrastive losses, which may not measure contribution to cross-modal retrieval. Meanwhile, more direct precedents, including LiT or ImageBind (-style method) are absent from the main comparison.
  • Under the experimental settings,the method present similar results with OGM-GE,which does not seem to prove the superiority of the dynamic allocation method in this article.

[1] Zhai, X. et al. “LiT: Zero-Shot Transfer with Locked-image Text Tuning.” CVPR, 2022.

[2] Girdhar, R. et al. “ImageBind: One Embedding Space to Bind Them All.” CVPR, 2023.

3 comments
AuthorsOpenReview ↗

Response to zEFG (Part I): Prior Work and the Scientific Value of Backward Allocation

We appreciate the reviewer's recognition of the encoder-specific admission mechanism. The connections to LiT, ImageBind, and OGM-GE are useful for clarifying how existing asymmetric training works and what new question BIA examines.

1. Mechanisms of asymmetric multimodal learning (W1, Q1)

These works provide important precedents for treating modalities differently. Their implementations, however, control different parts of training:

MethodImplementation and control signalWhere the control acts
LiT (2022)Locks a pretrained image tower and contrastively trains the text tower against its representations.Fixed image anchor; text-side learning.
ImageBind (2023)Freezes pretrained image/text encoders and uses image-paired contrastive objectives to train audio, depth, thermal, and IMU encoders into their shared space.Fixed reference encoders; newly aligned modalities.
OGM-GE (2022)Estimates per-modality contributions from correct-class predictions, attenuates the dominant modality's gradients, and adds Gaussian gradient noise for generalization.Modality-wide parameter gradients from the joint discriminative objective.
BIA (ours)Uses task–alignment gradient compatibility and relative norms to learn separate admissions for two trainable encoders.Only the shared alignment loss's backward gradient; branch-specific learning remains active.

OGM-GE is particularly relevant. In its original supervised formulation, modality-specific classifier logits yield correct-class confidence scores; their discrepancy determines which modality's gradient is reduced. Gaussian noise complements this modulation. For the retrieval comparison in our Appendix C, we adapted the strength estimate to exp⁡(−Lm)\exp(-L_m), using each branch's self-supervised loss, and retained modulation of branch gradients. BIA makes a different intervention within the same training objective: it isolates the shared contrastive alignment gradient and controls its admission independently of each branch's own gradient:

g~m=gmtask+λαmgmalign,m∈A,B.\tilde{g}_m=g_m^{\mathrm{task}}+\lambda\alpha_m g_m^{\mathrm{align}},\qquad m\in\\{A,B\\}.

The stop-gradient routing keeps the forward alignment value unchanged at a given parameter state. Both branch objectives continue updating their encoders, even when one alignment admission is zero. The learned policy responds to gradient interaction without fixing a teacher modality. We thank the reviewer for highlighting LiT in particular; we will add its citation and strengthen the mechanism-level discussion of ImageBind and OGM-GE in the revision.

2. Scientific findings behind the control dimension (W1, Q2)

The propositions define the control geometry: with βm=λαm\beta_m=\lambda\alpha_m, equal admissions lie on the scalar-equivalent diagonal βA=βB\beta_A=\beta_B, while asymmetric admission accesses encoder-side gradient combinations unavailable to a single scalar. The resulting scientific question is whether that extra freedom produces useful representation trade-offs beyond dense scalar tuning, and under which sensing conditions.

Our submitted SEN12 experiments test this question across five matched seeds. Comparing 44 scalar weights with 20 asymmetric allocations per seed yields 2.329% mean relative ID hypervolume expansion over cross-modal retrieval and two encoder-retention scores, with positive expansion in all five seeds. Under matched Clear/Cloudy sensing, the expansion is 0.083%/4.226%. The alignment-to-task gradient-norm ratios rise from 2.55 to 3.91 for one branch and 2.79 to 3.96 for the other under clouds, linking this difference to greater alignment pressure.

A direction-swap experiment identifies what the extra control changes. At λ=10\lambda=10, switching admissions from (0.25,1)(0.25,1) to (1,0.25)(1,0.25) gives nearly identical cross-modal R@5 (0.0983/0.0985), but changes the two retention scores from (0.3206, 0.2996) to (0.3054, 0.3104). Similar cross-modal retrieval can therefore coexist with different distributions of modality-specific retention.

The algebra establishes the available control space; these experiments show when it becomes valuable and what it changes in representation learning. This condition dependence motivates choosing admissions from the current optimization state. We will connect the gradient-level characterization, sensing-condition analysis, and retention findings more closely in the revision.

AuthorsOpenReview ↗

Response to zEFG (Part II): Matched ViT Validation and Retrieval Comparisons

We appreciate the reviewer's thoughtful assessment. The submission compares methods on a common retrieval task; your suggestions motivated more direct ViT controls and stronger validation-based selection.

3. From fixed allocations to a stronger matched ViT comparison (W2, Q3)

Our submitted SEN12 study establishes the representation trade-offs available through dense scalar tuning and fixed asymmetric allocation. The ViT-B/16 experiments examine how these controls translate into practical training policies. Following the reviewer's suggestion, we tested Scalar, two fixed asymmetric directions, and BIA-MLP on the same ViT-B/16 setup, with matched initialization, data, retrieval galleries, seven seeds, and training budgets (600/900 steps on DSEC/Sunboat).

At a common alignment weight of 5, BIA-MLP achieved 5.213% ID R@5 on DSEC, ahead of Scalar (4.363%) and the two fixed directions (4.615%/4.321%). Sunboat revealed a different pattern: a fixed direction achieved 1.716%, compared with 1.660% for Scalar and 1.231% for MLP. This result raised a useful follow-up question: would a fixed-strength comparison still favor the same approach once each training policy had been properly tuned?

We therefore expanded the comparison with independent validation-based selection. MLP and Scalar each received nine alignment-weight candidates; our Rule policy searched 15 settings, and Linear searched nine weights. All choices were made using development data and frozen before confirmation on seven new matched seeds (204–210). The resulting ID performance is:

ID R@5 (%)Tuned ScalarTuned RuleTuned LinearBIA-MLP
DSEC4.3533.9124.2164.636
Sunboat1.5111.5291.2501.734

After validation selection, BIA-MLP has the highest held-out mean on both datasets, exceeding tuned Scalar by 6.5% on DSEC and 14.8% on Sunboat. The initial fixed-weight test and this independent evaluation have separate numerical setups; each comparison is interpreted within its own protocol. Together, they show why allocation should be evaluated across training conditions and provide a closer ViT-level link between the original structural finding and the learned controller.

4. Retrieval adaptations and fair comparisons (W2, Q4)

The five optimization comparisons in submitted Table 1 use the same ViT-B/16 architecture, data splits, training-step budgets, and retrieval evaluator. This common task lets us examine their optimization ideas with consistent retrieval measurements. We appreciate the request to clarify adaptations and search budgets.

OGM-GE is a useful example. In its original classification setting, the contribution of each modality is estimated from correct-class predictions; the stronger modality's gradients are reduced, with Gaussian noise added during modulation. Our task has no class labels or fusion classifier, so the retrieval adaptation uses the branch self-supervised loss to estimate modality strength, s_m = exp(-L_m), while retaining the stronger-branch modulation and noise (Appendix C). BIA instead adjusts only the gradient contributed by the shared alignment loss, leaving each branch's self-supervised learning signal intact. We will detail this mapping and the relationship to LiT and ImageBind.

Our additional comparisons also include independently tuned Rule/Linear policies and SSA-Ret and MASAM-Ret, adaptations of two 2026 methods with parameters selected on development validation. Their DSEC/Sunboat ID R@5 means are 3.786%/1.585% and 4.049%/1.156%, respectively, compared with 4.636%/1.734% for BIA-MLP. We will document the adapters, tuning budgets, and compute costs.

5. Understanding the practical benefit relative to OGM-GE (W2, Q5)

The submitted five-seed results already show that the benefit varies across conditions. On DSEC, BIA-MLP improves ID R@5 from 3.49% with OGM-GE to 4.60%, a 31.8% relative gain. On Sunboat, the means are 1.64% and 1.67%, a smaller 1.8% gain. This motivates studying when unequal task–alignment gradient interactions make selective admission especially useful.

The additional experiments strengthen the evaluation of the selected controller. Under the new matched confirmation, BIA-MLP reaches 4.636% on DSEC and 1.734% on Sunboat, ahead of more extensively tuned Scalar by 6.5% and 14.8%. These results complement the submitted condition-dependent findings and help explain how alignment strength and admission policy jointly shape training.

We will integrate the ViT controls, adapter details, stronger selection, and condition-specific analysis into the revision. We would also welcome any further specific questions and would be glad to pursue targeted analyses or experiments that help strengthen the paper.

Dear Reviewer zEFG,

Thank you again for your valuable comments on our submission. We would like to kindly confirm whether we have sufficiently addressed all of your concerns (or at least part of them).

Should there be any remaining questions or areas requiring further clarification, please do not hesitate to let us know.

If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

Reviewer_avQJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates a limitation of conventional multimodal alignment: a single scalar coefficient jointly controls the alignment gradients entering different modality encoders, even when their information quality and optimization dynamics differ. The authors propose Backward Influence Allocation (BIA), introducing encoder-specific admission coefficients αA,αB∈[0,1]\alpha_A,\alpha_B\in[0,1] while keeping the forward alignment objective numerically unchanged:

g~m=gmtask+λαmgmalign.\tilde g_m=g_m^{\mathrm{task}}+\lambda\alpha_m g_m^{\mathrm{align}}.

The paper establishes that isotropic admission is equivalent to scalar alignment, whereas asymmetric admission enables gradient combinations that no single scalar coefficient can generally reproduce. Empirical characterization on SEN12 examines cross-modal retrieval, within-modality retention, and hypervolume coverage. The reported mean ID hypervolume expansion under cloudy sensing is 2.329% across five seeds, with considerably smaller gains under clear sensing.

The authors also introduce a lightweight controller that uses gradient compatibility and relative gradient magnitude to select admissions, trained through a REINFORCE-style objective based on training-only probe progress. ViT-B/16 experiments on DSEC, Eilat, and Sunboat show competitive performance against five specialized multimodal optimization baselines. Compared with validation-tuned scalar alignment, the controller improves mean ID R@5 on all three datasets, but the paired 95% confidence intervals include zero.

Overall, the paper provides a useful conceptual distinction between alignment strength and backward influence allocation. However, the evidence that the learned controller provides a reliable advantage over simpler allocation strategies remains limited.

Critical strengths

1. Clean formulation of encoder-specific alignment influence

The paper identifies backward influence allocation as a distinct optimization dimension and implements it through a simple stop-gradient gate without changing the numerical forward objective or adding inference-time modules. The isotropic equivalence and asymmetric non-reducibility results provide a clear formal characterization. This mechanism could be useful beyond the particular controller and contrastive objectives evaluated.

2. Informative empirical analysis of condition dependence

Section 4 goes beyond reporting a single retrieval score. A 44-weight scalar sweep and asymmetric configurations are compared over multiple seeds, with positive ID hypervolume expansion in all five cloudy SEN12 seeds. The clear-versus-cloudy comparison reports mean ID hypervolume expansion of 0.083% versus 4.226%, and reversed allocations produce different encoder-side retention at similar cross-modal retrieval. These results help identify when and why allocation may matter.

Critical weaknesses

1. The learned controller's advantage over scalar alignment is not statistically established

Table 1 reports the following ID R@5 results:

DatasetScalar (%)Ours (%)Difference (percentage points)
DSEC3.914.60+0.69
Eilat0.970.99+0.02
Sunboat1.381.67+0.29

The authors explicitly acknowledge that all paired 95% confidence intervals include zero. Validation performance is also mixed: the scalar baseline outperforms the controller on Eilat. With only five seeds and substantial variability, these experiments do not demonstrate a reliable practical improvement over scalar tuning. This matters because the learned controller is a major algorithmic contribution.

2. The necessity of learned gradient-conditioned allocation is insufficiently demonstrated

Appendix E shows that simpler linear and deterministic controllers can match or outperform the MLP in individual settings. At λ=5\lambda=5, the deterministic controller on DSEC achieves validation R@5 of 0.0866 versus 0.0552 for the MLP, and ID R@5 of 0.0426 versus 0.0377. On Sunboat, the deterministic controller also achieves higher ID retrieval (0.0170 versus 0.0135).

Moreover, a mean admission gap ∣αA−αB∣≈0.33|\alpha_A-\alpha_B|\approx0.33 demonstrates asymmetric actions, but not that the policy exploits the gradient state meaningfully; the paper notes that independent uniform sampling over ten levels yields an expected gap around 0.367. The training-probe reward also reflects evolving optimization progress rather than an isolated causal effect of the action. Without stronger controls, it is unclear whether the learned policy is necessary, or whether simple asymmetric or stochastic allocation explains most of the benefit.

Questions for authors

Q1. Can the authors demonstrate an advantage over simpler allocation strategies under matched model-selection and computational budgets? Appendix E shows competitive linear and deterministic controllers. Please compare the MLP, linear and deterministic policies, and fixed asymmetric admissions under identical seeds, alignment-weight selection, and compute budgets. This would clarify the incremental value of REINFORCE-based policy learning.

Q2. Does the policy learn meaningful dependence on gradient state? Please compare against a state-independent stochastic controller with matched marginal action frequencies. A state-shuffling ablation that preserves the learned action distribution would be particularly informative: a substantial performance decrease would support the claim that the controller exploits gradient interactions.

Q3. How robust are the improvements to more seeds and longer training? Appendix F reports paired ID confidence intervals of [−0.493,1.873][-0.493,1.873], [−0.525,0.575][-0.525,0.575], and [−0.797,1.371][-0.797,1.371] percentage points for DSEC, Eilat, and Sunboat, respectively. Can additional independent runs establish a reliable benefit? Do the gains persist with longer training, particularly on Eilat, where the fixed-step protocol corresponds to approximately one pass over the training set?

3 comments
AuthorsOpenReview ↗

Response to avQJ (Part I): Controller Selection and Gradient-State Dependence

Thank you for recognizing the contribution of encoder-specific backward allocation and for raising thoughtful questions about the learned controller. Your suggestions encouraged us to examine two closely related issues: how different admission policies compare after systematic selection, and how gradient-state information contributes to the learned decisions.

1. What does learned allocation offer beyond simpler policies? (W2, Q1)

BIA introduces independent control over the alignment gradients received by two trainable encoders. Rule, Linear, and MLP are different ways of selecting these admissions. Our original experiments established the additional allocation dimension; the expanded study examines how effectively each controller uses it under systematic model selection.

We conducted a new validation-selected comparison covering 15 Rule configurations and nine alignment weights each for Scalar, Linear, and MLP. Selection used three development seeds (201–203). The chosen configurations were frozen and evaluated on seven independent matched seeds (204–210), using the same ViT-B/16 architecture, data, initializations, encoder optimization steps, and retrieval evaluation.

Held-out ID R@5 (%)ScalarRuleLinearBIA-MLP
DSEC4.3533.9124.2164.636
Sunboat1.5111.5291.2501.734

MLP achieves the highest held-out mean on both datasets. It improves over tuned Scalar by 6.5% on DSEC and 14.8% on Sunboat, with additional gains over the independently selected Rule and Linear policies. These results provide a clearer empirical basis for choosing the learned controller.

We also evaluated fixed asymmetric admissions in a separate ViT experiment. The preferred fixed direction changes between DSEC and Sunboat, reinforcing our finding that effective backward allocation depends on the sensing condition. The revised Appendix E presents these comparisons alongside the controller-selection study.

For computational transparency, we report the matched encoder-update budgets and controller training costs. MLP introduces only 3,060 training-time parameters, with no additional inference-time components. The expanded comparisons and computational details are provided in Appendices B, E, and F.

2. How does gradient-state information affect the learned policy? (W2, Q2)

The original admission analysis showed that MLP learns asymmetric allocations. We extended this analysis to examine how those decisions depend on the observed gradient state and their timing during training.

First, we introduced two direct state interventions. StateBlind removes the four gradient-state inputs, and StateShuffle disrupts their correspondence with the current training state while retaining the policy-learning procedure.

On DSEC, the normal controller achieves 4.508% ID R@5, compared with 4.097% for StateBlind and 3.789% for StateShuffle. The Normal–StateShuffle difference reaches +0.720 pp, with a paired 95% confidence interval of [+0.039, +1.400] pp. This provides direct evidence that gradient-state information contributes to the learned allocation on DSEC.

Second, we investigated whether the observed effect could be explained solely by the frequency of selected actions. We performed 42 action-replay runs, preserving the complete joint-action multiset of (αA,αB)(\alpha_A,\alpha_B) for each source run while changing its temporal ordering. This matches the frequencies of entire paired actions, including their joint occurrences.

Ordered replay reproduces the original validation and ID results across all 14 source cases. Global shuffling changes the training outcomes:

ID R@5 (%)Ordered replayGlobal shuffle
DSEC4.6364.206
Sunboat1.7341.343

The state interventions and frequency-matched replay reveal complementary aspects of the controller: gradient information contributes to its decisions, and admission timing shapes the resulting optimization trajectory. The training reward supplies the policy-learning signal; these additional interventions examine the resulting behavior directly.

We have incorporated these findings into Section 5.4, Figure 6, and Appendix E, providing a more detailed account of how learned backward allocation operates. Part II addresses the independent statistical confirmation and longer-training evaluation.

AuthorsOpenReview ↗

Response to avQJ (Part II): Independent Confirmation and Longer-Horizon Evidence

Thank you again for your suggestions concerning statistical reliability and longer training. We extended the study with independent confirmation runs on DSEC and Sunboat and a longer-horizon Eilat experiment. These evaluations provide additional evidence about controller performance and how backward allocation behaves as training progresses.

3. Independent confirmation across additional seeds (W1, Q3)

Our submitted SEN12 study established that independent admissions expose representation trade-offs beyond dense scalar tuning, with 2.329% mean relative ID hypervolume expansion across five seeds. We have now complemented this structural evidence with a more extensive evaluation of the learned controller on retrieval tasks.

Following validation-only selection, we evaluated frozen Scalar, Rule, Linear, and MLP configurations on seven independent matched seeds (204–210) for DSEC and Sunboat. The encoder architecture, data, initialization, training steps, and evaluation protocol were matched within each dataset.

BIA-MLP achieves the highest mean ID R@5 among all four strategies on both datasets. Relative to independently tuned Scalar, its mean improvements are:

  • DSEC: +0.283 pp (+6.5% relative)
  • Sunboat: +0.224 pp (+14.8% relative)

The paired 95% confidence intervals are [−1.155, +1.721] pp and [−0.204, +0.652] pp, respectively. The complete seed-level statistics are reported in Appendix F.

These independent results extend our original experiments in two ways. First, they confirm the favorable mean performance of MLP after broader validation-based selection, using new seeds excluded from model selection. Second, they strengthen the connection between the structural flexibility established in Section 4 and the practical use of learned admissions in Section 5.

4. Eilat performance across longer training horizons (Q3)

Your observation about Eilat's training duration prompted us to examine the effect of additional training exposure directly.

The submitted Eilat protocol uses 700 optimizer steps, corresponding to approximately one pass over its training set. We conducted a separate experiment extending BIA-MLP and Scalar to 7,000 steps, using matched initializations and the same alignment weight, λ=5\lambda=5.

We recorded Validation R@5 at five checkpoints along each training trajectory, covering approximately 1.0, 2.5, 5.1, 7.2, and 10.1 training-set passes.

Training stepsBIA-MLP Val R@5 (%)Scalar Val R@5 (%)
7001.9091.957
1,7502.2262.297
3,5002.0601.893
5,0001.7821.473
7,0001.9171.782

A particularly interesting finding is that the relative ordering changes with training exposure. Scalar leads at the first two checkpoints, while BIA-MLP achieves higher mean Validation R@5 at 3,500, 5,000, and 7,000 steps. At 7,000 steps, MLP reaches 1.917%, compared with 1.782% for Scalar.

The curves also reveal non-monotonic retrieval behavior as both methods receive additional data exposure. This provides a more detailed understanding of the interaction between training duration and backward allocation.

The experiment uses two matched seeds, with all checkpoints recorded from the same 7,000-step trajectories. We have included the full curves in Appendix F.6, Figure 8, distinguishing this long-horizon validation analysis from the original five-seed ID evaluation.

Together, the expanded controller confirmation and Eilat training analysis provide stronger empirical support for the paper's central finding: backward influence allocation introduces a meaningful, condition-dependent optimization dimension, and its effectiveness depends on how admissions are selected and applied throughout training.

We appreciate your careful questions, which helped us develop these additional analyses. We hope the new evidence and revised manuscript address your concerns. Would you kindly let us know whether the remaining questions have been resolved, or whether any specific point would benefit from further discussion?

Dear Reviewer avQJ,

Thank you again for your valuable comments on our submission. We would like to kindly confirm whether we have sufficiently addressed all of your concerns (or at least part of them).

Should there be any remaining questions or areas requiring further clarification, please do not hesitate to let us know.

If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Author Summary and Sincere Appreciation

As the discussion period comes to a close, we would like to briefly summarize the contribution of our work, the improvements made during the review process, and our sincere appreciation for everyone who has helped us along the way.

The central contribution of this paper is to identify an overlooked asymmetry in multimodal alignment. Even when two encoders share the same alignment objective, its gradients can influence their representations differently. Backward Influence Allocation (BIA) introduces independent, state-conditioned gradient admission for each encoder, establishing a distinct dimension of control over alignment's backward influence. Our structural experiments show how this distinction affects encoder-side representation retention under heterogeneous sensing, while our learned controller demonstrates how it can be used in practical multimodal training.

The review process has substantially strengthened this work, both scientifically and in its presentation. Guided by the reviewers' questions, we revisited the fairness of our comparisons, expanded controller and baseline evaluations, incorporated a LiT-style comparison directly into Table 1, and added independent multi-seed confirmation with additional recent methods. We also introduced fixed-direction controls, gradient-state interventions, joint-action sequence replay, and extended training-exposure analysis. Beyond the experiments, we refined the methodological explanations, strengthened the discussion of our findings, and redesigned several key figures and their captions to make the underlying ideas and evidence easier to follow. These revisions have made the paper more rigorous, complete, and accessible. For ease of review, we have uploaded the revised manuscript and included a separately highlighted version in the supplementary materials, making the substantive changes easier to identify.

We feel genuinely fortunate to have received such thoughtful and responsible reviews. Each reviewer approached our work from a different perspective, raised questions that deserved careful investigation, and offered constructive suggestions that helped us improve the research itself. We are especially grateful for the attention, patience, encouragement, and recognition shown throughout our exchanges. It has been a privilege to engage with reviewers who take both scientific rigor and constructive dialogue so seriously.

Finally, we sincerely thank all three reviewers, our Area Chair, the Senior Program Chairs, and everyone involved in the review process. We deeply appreciate the considerable time, expertise, and care that go into evaluating and improving research, particularly at a conference of this scale.

Thank you all for your thoughtful contributions to this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.