- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates backward influence allocation as an additional control dimension in heterogeneous multimodal alignment, beyond the conventional scalar alignment strength. The central idea is to introduce encoder-specific admission coefficients that independently regulate how the gradient of a shared alignment objective enters each modality encoder, while keeping the forward alignment objective unchanged.
Overall, the paper presents a conceptually interesting perspective on multimodal optimization.
Critical strengths
- A clear and potentially useful optimization perspective on multimodal alignment.
- The effort to investigate when and why backward allocation matters. This transparency strengthens the credibility of the empirical analysis.
- The narrative is clear and the language is well-articulated.
Critical weaknesses
-
Insufficient evidence that the learned controller acquires a meaningful state-dependent allocation policy. The main empirical contribution is the proposed learned controller, but the current experiments do not convincingly isolate the benefit of policy learning from the benefit of introducing additional stochastic or time-varying alignment coefficients.
-
Table 9 shows that a simple rule-based controller achieves competitive performance and outperforms the learned MLP in several settings. These observations leave open whether the REINFORCE-based learning procedure provides a substantial advantage over simpler allocation strategies.
-
Most of the evaluated baselines were proposed between 2020 and 2024, with DGL (2025) being the most recent. The authors are encouraged to include representative state-of-the-art methods from 2026, particularly those addressing modality imbalance and gradient interaction, under comparable experimental settings.
-
Limited statistical support and incomplete evidence of practical superiority. The current results establish competitiveness, but not reliable superiority over carefully tuned scalar alignment.
-
Table 11 shows that at an alignment weight of 5, the MLP controller underperforms scalar alignment on all three ID benchmarks. Can the authors explain this behavior and clarify under which conditions the learned controller is expected to be advantageous?
-
Figure 5 requires substantial improvement in presentation quality. In the rightmost panel (“Backward allocation”), the arrows associated with (gA) and (gB) appear to originate from disconnected points, with no clear connection to their corresponding gradient sources. Moreover, several gradient labels contain vertically stacked superscripts, making the mathematical notation difficult to parse.
Questions for authors
Please see the critical weaknesses above. If the authors can convincingly address these concerns, I would be willing to reconsider and potentially raise my score.
6 comments
Response to WGwy (Part II): Recent Methods, Statistical Reliability, and Alignment Sensitivity
We appreciate the reviewer's careful suggestions. They encouraged us to extend the comparisons, strengthen independent confirmation, and examine how alignment strength interacts with encoder-specific admission.
3. Extending the comparison to recent methods (W3)
The submitted Table 1 compares five multimodal optimizers, including OGM-GE and DGL. We added two recent approaches: Shared Space Alignment (SSA; ICML 2026), which studies shared cross-modal representations, and MASAM (ICLR 2026), which uses modality-aware sharpness-aware updates.
We adapted their mechanisms to the common ViT-B/16 dual-encoder retrieval setting (SSA-Ret and MASAM-Ret). Within each dataset, the methods share data manifests, seed-matched initialization, training-step budgets, and ID evaluation. Method-specific settings were selected using development validation.
| ID R@5 (%) | BIA-MLP | SSA-Ret | MASAM-Ret |
|---|---|---|---|
| DSEC (7 seeds) | 4.636 | 3.786 | 4.049 |
| Sunboat (7 seeds) | 1.734 | 1.585 | 1.156 |
BIA-MLP achieves the highest mean on both datasets. We will explain the adaptations and computational costs in the revision, making the connections between these optimization mechanisms and backward allocation clear.
4. Statistical reliability and structural value (W4)
The submitted SEN12 experiments examined whether encoder-specific backward influence reveals trade-offs beyond dense scalar tuning. Across five seeds, 44 scalar weights and 20 asymmetric allocations per seed produced 2.329% mean relative ID hypervolume expansion. Under matched Clear/Cloudy conditions, expansion was 0.083%/4.226%, revealing a stronger effect when sensing quality deteriorates.
We also strengthened the learned-controller evaluation. Hyperparameters were selected exclusively on development seeds 201–203, then frozen for seven independent, seed-matched confirmations (204–210). Both methods received their own validation-selected settings.
| Held-out ID R@5 (%) | BIA-MLP | Tuned Scalar | Relative gain |
|---|---|---|---|
| DSEC | 4.636 | 4.353 | +6.5% |
| Sunboat | 1.734 | 1.511 | +14.8% |
The expanded confirmation shows higher mean retrieval for BIA-MLP against a stronger, independently tuned Scalar. Together with the SEN12 frontier results, it clarifies both the additional optimization choices opened by BIA and the performance of our selected learned realization.
5. What the fixed-λ results reveal about policy selection (W5)
The reviewer's observation about Table 11 raises an interesting question: how does the overall alignment weight affect the value of a learned admission policy? For encoder m, the effective alignment influence is βm = λαm. The weight λ sets its scale, while the learned admissions distribute it between encoders. We therefore revisited the fixed-λ setting through more thorough validation-based selection.
| Dataset | Submitted λ=5: MLP / Scalar | Submitted selected MLP | Relative change to new held-out: MLP / Scalar |
|---|---|---|---|
| DSEC | 3.49 / 3.96 | λ=2.5: 4.60 | +32.8% / +9.9% |
| Sunboat | 1.23 / 1.46 | λ=10: 1.67 | +41.0% / +3.5% |
All scores in the first two columns are ID R@5 (%). The last column compares each submitted fixed-λ=5 result with its new held-out mean in Section 4. It describes changes across two experimental protocols; the matched comparison is reported above.
In the new study, we searched nine λ values on development seeds 201–203, froze the chosen settings, and evaluated them on seeds 204–210. The tuned Scalar also improves over its original fixed-λ reference, while BIA-MLP shows a substantially larger relative increase. More importantly, BIA-MLP outperforms the separately tuned Scalar by 6.5% on DSEC and 14.8% on Sunboat within the same independent confirmation protocol. This provides a clearer account of how alignment strength and encoder-specific admission interact, and why policy selection matters.
6. Clarifying the gradient paths in Figure 5 (W6)
We thank the reviewer for this careful observation. It helps us improve the figure's precision and make the method easier to follow. We will redraw the rightmost panel with two alignment-gradient paths traced from the shared alignment loss to the corresponding encoders, label them directly with αA and αB, and show branch-specific task gradients separately. We will also correct the superscripts and harmonize notation. The caption will state the mechanism clearly: the forward alignment objective remains unchanged, while its backward influence is allocated independently to each encoder.
We will incorporate these results and improve the explanation of the connection between BIA's structural flexibility, learned policy selection, and sensitivity to alignment strength.
Follow-up on the Rebuttal
I thank the authors for their detailed rebuttal. I would appreciate further clarification on three remaining points:
-
With an ID gallery of 383 samples, random retrieval yields an expected R@5 of approximately 1.3%, whereas BIA-MLP achieves 1.734% and tuned Scalar achieves 1.511%. The absolute improvement is therefore only 0.223 pp. Could the authors elaborate on the practical significance of this gain and whether the results provide sufficiently strong evidence of meaningful representation learning on this dataset?
-
The additional hyperparameter search convincingly demonstrates that selecting λ is important. However, this does not fully explain the original observation in Table 11: at the same λ = 5, MLP underperforms Scalar on all three ID benchmarks. Could the authors provide a explanation for this behavior?
-
At the time of writing this follow-up, the PDF available to reviewers still appears unchanged, for example, Fig. 5.
If I have misunderstood anything, please feel free to correct me.
Response to WGwy (Part I): State Dependence and Policy Selection
We thank the reviewer for recognizing the backward-allocation perspective. Our paper first identifies encoder-specific backward influence as a new control dimension, then proposes Rule-τ, Linear-K10, and MLP as alternative realizations. We selected MLP for the main study. These are our own proposed strategies, not external baselines. Your concerns motivated two targeted tests: whether MLP uses gradient state and whether its selection holds up after stronger tuning of the alternatives.
1. Does the MLP learn meaningful state-dependent allocation (W1)?
The submission shows asymmetric MLP actions, but action asymmetry alone cannot establish useful state dependence. Keeping the MLP, REINFORCE, and action grid fixed, we intervened on its four gradient-state features (two compatibilities, two log norm ratios). StateBlind zeros the inputs; StateShuffle breaks their temporal correspondence with training. Both preserve stochastic policy learning.
Across five matched held-out seeds (204–208), MLP exceeds StateBlind/StateShuffle on DSEC ID R@5 by +0.411/+0.720 percentage points. Against StateShuffle it wins in four seeds and ties in one (paired 95% CI [+0.039, +1.400] pp). Sunboat gaps are +0.131/0.000 pp: the benefit of state information is condition-dependent.
To control for action frequency, we ran 42 action-matched replays (seven seeds per dataset × three variants): original order, global permutation, or permutation within consecutive eight-action blocks. Each uses the identical multiset of joint (αA, αB) actions per seed. Ordered replay exactly reproduces the original validation and ID R@5 in all 14 cases.
Global shuffling lowers mean ID R@5 by 0.430 pp (DSEC) and 0.392 pp (Sunboat) despite matched actions; local shuffling has mixed effects. State interventions test gradient-information use directly, while replay shows that action timing also matters. These controls go beyond the original admission-gap diagnostic.
2. Reassessing our proposed allocation strategies under stronger tuning (W2)
In the submitted work, Rule deterministically maps gradient statistics to admissions; Linear and MLP learn from those statistics using linear and nonlinear policies. All three are implementations we proposed for BIA. MLP was selected for full-scale evaluation before final test results. Table 9 showed that multiple implementations are viable, but at the shared λ=5, Rule sometimes exceeded MLP. We therefore retested their relative performance after stronger tuning.
We searched 15 (τ, λ) Rule configurations, nine Linear λ values, and MLP's existing nine-weight grid, selecting only on development Validation (seeds 201–203). Frozen choices were tested on seven independent, matched seeds (204–210), with the same ViT-B/16 initialization, splits, optimizer steps, and ID evaluator. Scalar is included as the conventional non-allocation reference.
After stronger tuning, MLP has the highest held-out mean on both datasets. Its paired gains over Rule/Linear are +0.724/+0.420 pp (DSEC) and +0.205/+0.485 pp (Sunboat), winning in 6/7, 5/7 DSEC seeds and 4/7, 5/7 Sunboat seeds, respectively. Paired 95% CIs include zero; these are mean-level gains, not proven universal superiority.
Crucially, Rule’s occasional advantage in Table 9 does not persist in the validation-selected, held-out mean comparison on DSEC or Sunboat. This supports our choice of the MLP while keeping the broader contribution clear: BIA opens an optimization dimension with several viable realizations. We will explain the state tests and frozen selection protocol in the revision to distinguish the allocation principle from the merits of our chosen implementation.