- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies gate degeneration in memory-augmented VLAs, where the memory projection compensates for gate values through scale coupling. It proposes BIND-VLA, which orthogonalizes and normalizes the memory update direction, allowing the gate to control a bounded relative update magnitude. With compressed external memory, BIND-VLA achieves 38.4% success on nine MIKASA-Robo Remember tasks, versus 32.4% for dot-product gating, while preserving memory-free performance on LIBERO. Ablations and inference-time interventions support the importance of adaptive update magnitudes.
Critical strengths
- The paper explains why gate values alone cannot characterize injection strength and introduces a fusion rule with an explicit magnitude bound. This connects the identified failure to a precise architectural remedy.
- Same-backbone comparisons, normalization and orthogonalization ablations, and gate interventions support the proposed mechanism.
Critical weaknesses
Initialization differences limit the interpretation of the Dot + Ortho ablation. Its initial relative update is approximately 0.35, compared with 0.033 for BIND-VLA (Appendix C.3). Without controlling this difference, its poor performance cannot be clearly attributed to the gate design rather than excessive initial perturbation. This weakens the conclusion drawn from this particular ablation.
Questions for authors
- Do the conclusions hold when gate initialization and initial update magnitude are matched across fusion variants, particularly for Dot + Ortho?
- Are the main performance gains and gate-degeneration patterns consistent across independent training seeds?
- How does BIND-VLA’s scale-coupling diagnosis and proposed remedy relate to GMP’s analysis of continuous-gate saturation and calibrated gating?
5 comments
Q2. Consistency across training seeds.
Yes, both are consistent. BIND-VLA is the best design under every seed, and the gates degenerate in the same way under every seed.
We retrained the five main configurations with two additional training seeds, keeping all other settings identical (seed 1 is the run in the paper). Within a seed, all configurations share the same data order, and all runs are evaluated on the same episodes, so configurations can be compared in pairs.
Performance. Average success rate (%) on the nine Remember tasks.
| Configuration | Seed 1 | Seed 2 | Seed 3 | Mean ± Std | Gap to BIND-VLA | 95% CI of gap |
|---|---|---|---|---|---|---|
| No memory | 15.4 | 12.1 | 18.2 | 15.2 ± 3.1 | 23.2 ± 1.4 | [19.8, 26.6] |
| Dot | 32.4 | 29.8 | 34.7 | 32.3 ± 2.5 | 6.2 ± 0.8 | [4.3, 8.1] |
| MLP | 28.1 | 24.7 | 30.6 | 27.8 ± 3.0 | 10.7 ± 1.3 | [7.5, 13.9] |
| MLP + Unit | 31.8 | 28.8 | 34.5 | 31.7 ± 2.9 | 6.8 ± 1.2 | [3.9, 9.6] |
| BIND-VLA | 38.4 | 36.8 | 40.2 | 38.5 ± 1.7 | – | – |
Std is the sample standard deviation over three seeds. Gaps are computed seed by seed, and the interval is a paired -interval ().
Individual configurations vary by 1.7–3.1 points, but they rise and fall together, so the paired gaps are much more stable (std 0.8–1.4 points), and every paired interval excludes zero.
Gate degeneration. We probed the checkpoints of all three seeds as in Figure 1.
| Mean gate | Seed 1 | Seed 2 | Seed 3 |
|---|---|---|---|
| Dot | 0.999 | 0.999 | 0.999 |
| MLP | 0.0067 | 0.0065 | 0.0072 |
| BIND-VLA | 0.081 | 0.093 | 0.078 |
Under every seed, the Dot gate saturates and the MLP gate collapses, whereas the gate of BIND-VLA stays at a moderate level, with its update set by . The new controls in W1 & Q1 use a single seed.
Revision. New Appendix J (Table 18 and the paragraph on gate statistics), with a pointer in Sec. 5.2.
Q3. Relation to GMP.
The two works are complementary. GMP [1] explains in which direction a gate drifts and decides when memory is used. Our diagnosis concerns whether the gate value determines the injection strength at all, and our remedy sets how much memory each token receives. We now cite and discuss GMP.
| GMP | BIND-VLA | |
|---|---|---|
| Observation | Jointly trained gate tends toward 1, or toward 0 under a strong gate penalty | Gate saturates or collapses while the projection compensates |
| Explanation | Training objective: more history lowers training error | Scale coupling: training constrains only the product of gate and projection |
| Question answered | When is memory used? | Does the gate value set the injection strength? |
| Remedy | Binary gate per timestep, calibrated with labels from a validation split | Decoupled fusion, , learned end to end per token |
Diagnosis. GMP's observations mirror the two failure modes we report. GMP adds the gated history readout to the residual stream, the same form as our Eq. 1. Its gate thus decides when memory is used, but once it is open, the injection strength is left to the memory branch and is neither modulated nor bounded by the gate. With a continuous gate, this form further lets the projection compensate for the gate, as for our MLP gate, whose mean of 0.0067 still yields a relative update of 0.062.
Remedy. The two could be combined. Under our fusion rule, an open GMP gate would yield a bounded update whose strength is still adapted per token.
Revision. GMP is now cited in Related Work (Sec. 2) and discussed in the new Appendix K.
We thank the reviewer again for the constructive feedback. The matched-initialization control in particular closes a gap in our original ablation. We hope the responses resolve the remaining concerns, and we would be glad to discuss any further questions.
[1] Yihuai Gao, Jinyun Liu, Shuang Li, and Shuran Song. Gated Memory Policy. arXiv preprint arXiv:2604.18933, 2026.
Dear Reviewer 1tyc,
We are sorry to bother you again. As the discussion period closes in two days, we would like to check whether our response and revision have resolved your concerns. We would greatly appreciate any feedback, and we remain happy to answer further questions.
Best regards, The Authors
We thank the reviewer for the careful reading, in particular for spotting the uncontrolled initialization of Dot + Ortho. All points are addressed with new experiments, included in the revised PDF (changes in blue).
New since submission: (1) Dot + Ortho retrained with its initial update matched to BIND-VLA, (2) MLP retrained with the initialization of BIND-VLA, (3) BIND-VLA retrained with a single globally learned gate, (4) three training seeds for the main configurations, and (5) a discussion of GMP.
W1 & Q1. Initialization of Dot + Ortho, and conclusions under matched initialization.
We agree that the original ablation did not control this difference. With the initial update matched exactly, Dot + Ortho is unchanged (11.2% vs. 11.3%), so its failure stems from the Dot gate rather than from the initial perturbation.
Dot + Ortho with matched initialization. The Dot gate has no bias, so g≈0.5 at initialization. We add a fixed, non-learnable bias of −3 to its logit, so that the initial gate is σ(−3)≈0.047 and the initial relative update is ρσ(−3)≈0.033, exactly as in BIND-VLA. All other settings are unchanged. Since the bias is not trained, it only moves the starting point and gives the gate no additional degree of freedom.
The gate still saturates at 0.99 and the update is pushed close to the full budget. The logit of the Dot gate scales with D (Eq. 22), so a fixed offset is quickly overwhelmed. Once the gate is saturated, the gradient reaching the logit is scaled by g(1−g) and nearly vanishes, and the injection stays near ρ on every token, about twelve times the update learned by BIND-VLA from the same starting point.
Other fusion variants. MLP + Unit already shares the initialization of BIND-VLA. We additionally retrained MLP and BIND-VLA with a global learned gate from the same initialization (b2=−3, truncated-normal WO).
For the three decoupled variants, both the initial gate and the initial update (0.033) are matched exactly. For MLP, the gate and projection initializations are matched, but under residual fusion its initial update also depends on the projection scale. The Dot and MLP gates still degenerate, and all variants remain below BIND-VLA.
Too small an initial update is not the cause either. Dot, Scaled Dot, MLP and MLP + Norm start from WO=0 and thus from a zero initial update, yet their gates still degenerate (Sec. 3). The degeneration therefore appears with a zero, a matched and a large initial update.
Revision. New Appendix I.2 and I.3 (Tables 16–17), with pointers in Sec. 5.2 and Appendix C.3.