acceptodds
Under review as a conference paper at ICLR 2027

When the Gate Does Not Gate: Bounded Memory Injection with Normalized Direction for Vision-Language-Action Models

Abstract

Memory-dependent manipulation requires robots to act on visual cues that are no longer observable. Existing memory-augmented vision-language-action (VLA) models retrieve history from an external memory bank and add it to the current representation as a gated residual. The gate is expected to control how strongly memory is injected. However, we show that this expectation often fails. Across common designs, the gate degenerates. It either saturates or collapses toward zero, losing its variation across inputs. Control over injection strength then shifts to the memory projection, which undermines the fusion of memory. We trace this to scale coupling between the gate and the projection. Because training constrains only their product, the gate value neither reflects nor bounds injection strength. We propose BIND-VLA, which decouples the direction and magnitude of the memory update. The projection is orthogonalized against the current representation and normalized, so it determines only the update direction. The gate sets the fraction of a fixed update budget. It thus becomes identifiable, and memory injection is explicitly bounded. BIND-VLA also compresses each past timestep into a single memory token and keeps history outside the backbone, leaving the input length unchanged. On nine MIKASA-Robo memory tasks, BIND-VLA reaches 38.4% average success, 6.0 points above dot-product gated fusion with the same architecture. Its gate no longer degenerates, with a coefficient of variation of about 50% compared with below 5% for the dot-product gate. Fixing it to its mean value lowers average success by 9.2 points. On LIBERO, the same gate switches memory off when history is not needed, and BIND-VLA matches the memory-free policy at 92.4% versus 92.2%.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Memory-Augmented Policies, Gated Fusion, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_1tycOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies gate degeneration in memory-augmented VLAs, where the memory projection compensates for gate values through scale coupling. It proposes BIND-VLA, which orthogonalizes and normalizes the memory update direction, allowing the gate to control a bounded relative update magnitude. With compressed external memory, BIND-VLA achieves 38.4% success on nine MIKASA-Robo Remember tasks, versus 32.4% for dot-product gating, while preserving memory-free performance on LIBERO. Ablations and inference-time interventions support the importance of adaptive update magnitudes.

Critical strengths
  1. The paper explains why gate values alone cannot characterize injection strength and introduces a fusion rule with an explicit magnitude bound. This connects the identified failure to a precise architectural remedy.
  2. Same-backbone comparisons, normalization and orthogonalization ablations, and gate interventions support the proposed mechanism.
Critical weaknesses

Initialization differences limit the interpretation of the Dot + Ortho ablation. Its initial relative update is approximately 0.35, compared with 0.033 for BIND-VLA (Appendix C.3). Without controlling this difference, its poor performance cannot be clearly attributed to the gate design rather than excessive initial perturbation. This weakens the conclusion drawn from this particular ablation.

Questions for authors
  1. Do the conclusions hold when gate initialization and initial update magnitude are matched across fusion variants, particularly for Dot + Ortho?
  2. Are the main performance gains and gate-degeneration patterns consistent across independent training seeds?
  3. How does BIND-VLA’s scale-coupling diagnosis and proposed remedy relate to GMP’s analysis of continuous-gate saturation and calibrated gating?
5 comments

We thank the reviewer for the careful reading, in particular for spotting the uncontrolled initialization of Dot + Ortho. All points are addressed with new experiments, included in the revised PDF (changes in blue).

New since submission: (1) Dot + Ortho retrained with its initial update matched to BIND-VLA, (2) MLP retrained with the initialization of BIND-VLA, (3) BIND-VLA retrained with a single globally learned gate, (4) three training seeds for the main configurations, and (5) a discussion of GMP.

W1 & Q1. Initialization of Dot + Ortho, and conclusions under matched initialization.

We agree that the original ablation did not control this difference. With the initial update matched exactly, Dot + Ortho is unchanged (11.2% vs. 11.3%), so its failure stems from the Dot gate rather than from the initial perturbation.

Dot + Ortho with matched initialization. The Dot gate has no bias, so g≈0.5g \approx 0.5 at initialization. We add a fixed, non-learnable bias of −3-3 to its logit, so that the initial gate is σ(−3)≈0.047\sigma(-3) \approx 0.047 and the initial relative update is ρ σ(−3)≈0.033\rho\,\sigma(-3) \approx 0.033, exactly as in BIND-VLA. All other settings are unchanged. Since the bias is not trained, it only moves the starting point and gives the gate no additional degree of freedom.

SettingInitial uuGate after traininguu after trainingAvg. success (%)
Dot + Ortho (original)0.350.990.6911.3
Dot + Ortho (matched initialization)0.0330.990.6911.2
BIND-VLA0.0330.0810.05738.4

The gate still saturates at 0.99 and the update is pushed close to the full budget. The logit of the Dot gate scales with D\sqrt{D} (Eq. 22), so a fixed offset is quickly overwhelmed. Once the gate is saturated, the gradient reaching the logit is scaled by g(1−g)g(1-g) and nearly vanishes, and the injection stays near ρ\rho on every token, about twelve times the update learned by BIND-VLA from the same starting point.

Other fusion variants. MLP + Unit already shares the initialization of BIND-VLA. We additionally retrained MLP and BIND-VLA with a global learned gate from the same initialization (b2=−3b_2=-3, truncated-normal WOW_O).

Variant (BIND-VLA initialization)Gate after trainingSuccess (%)
Dot + OrthoSaturated (0.99)11.2
MLPCollapsed (2.6×10−42.6\times10^{-4})17.6
MLP + UnitInput-dependent31.8
BIND-VLA, global learned gateConstant33.2
BIND-VLAInput-dependent38.4

For the three decoupled variants, both the initial gate and the initial update (0.033) are matched exactly. For MLP, the gate and projection initializations are matched, but under residual fusion its initial update also depends on the projection scale. The Dot and MLP gates still degenerate, and all variants remain below BIND-VLA.

Too small an initial update is not the cause either. Dot, Scaled Dot, MLP and MLP + Norm start from WO=0W_O = 0 and thus from a zero initial update, yet their gates still degenerate (Sec. 3). The degeneration therefore appears with a zero, a matched and a large initial update.

Revision. New Appendix I.2 and I.3 (Tables 16–17), with pointers in Sec. 5.2 and Appendix C.3.

Q2. Consistency across training seeds.

Yes, both are consistent. BIND-VLA is the best design under every seed, and the gates degenerate in the same way under every seed.

We retrained the five main configurations with two additional training seeds, keeping all other settings identical (seed 1 is the run in the paper). Within a seed, all configurations share the same data order, and all runs are evaluated on the same episodes, so configurations can be compared in pairs.

Performance. Average success rate (%) on the nine Remember tasks.

ConfigurationSeed 1Seed 2Seed 3Mean ± StdGap to BIND-VLA95% CI of gap
No memory15.412.118.215.2 ± 3.123.2 ± 1.4[19.8, 26.6]
Dot32.429.834.732.3 ± 2.56.2 ± 0.8[4.3, 8.1]
MLP28.124.730.627.8 ± 3.010.7 ± 1.3[7.5, 13.9]
MLP + Unit31.828.834.531.7 ± 2.96.8 ± 1.2[3.9, 9.6]
BIND-VLA38.436.840.238.5 ± 1.7––

Std is the sample standard deviation over three seeds. Gaps are computed seed by seed, and the interval is a paired tt-interval (n=3n=3).

Individual configurations vary by 1.7–3.1 points, but they rise and fall together, so the paired gaps are much more stable (std 0.8–1.4 points), and every paired interval excludes zero.

Gate degeneration. We probed the checkpoints of all three seeds as in Figure 1.

Mean gateSeed 1Seed 2Seed 3
Dot0.9990.9990.999
MLP0.00670.00650.0072
BIND-VLA0.0810.0930.078

Under every seed, the Dot gate saturates and the MLP gate collapses, whereas the gate of BIND-VLA stays at a moderate level, with its update set by ρg\rho g. The new controls in W1 & Q1 use a single seed.

Revision. New Appendix J (Table 18 and the paragraph on gate statistics), with a pointer in Sec. 5.2.

Q3. Relation to GMP.

The two works are complementary. GMP [1] explains in which direction a gate drifts and decides when memory is used. Our diagnosis concerns whether the gate value determines the injection strength at all, and our remedy sets how much memory each token receives. We now cite and discuss GMP.

GMPBIND-VLA
ObservationJointly trained gate tends toward 1, or toward 0 under a strong gate penaltyGate saturates or collapses while the projection compensates
ExplanationTraining objective: more history lowers training errorScale coupling: training constrains only the product of gate and projection
Question answeredWhen is memory used?Does the gate value set the injection strength?
RemedyBinary gate per timestep, calibrated with labels from a validation splitDecoupled fusion, u=ρgu=\rho g, learned end to end per token

Diagnosis. GMP's observations mirror the two failure modes we report. GMP adds the gated history readout to the residual stream, the same form as our Eq. 1. Its gate thus decides when memory is used, but once it is open, the injection strength is left to the memory branch and is neither modulated nor bounded by the gate. With a continuous gate, this form further lets the projection compensate for the gate, as for our MLP gate, whose mean of 0.0067 still yields a relative update of 0.062.

Remedy. The two could be combined. Under our fusion rule, an open GMP gate would yield a bounded update whose strength is still adapted per token.

Revision. GMP is now cited in Related Work (Sec. 2) and discussed in the new Appendix K.

We thank the reviewer again for the constructive feedback. The matched-initialization control in particular closes a gap in our original ablation. We hope the responses resolve the remaining concerns, and we would be glad to discuss any further questions.

[1] Yihuai Gao, Jinyun Liu, Shuang Li, and Shuran Song. Gated Memory Policy. arXiv preprint arXiv:2604.18933, 2026.

Dear Reviewer 1tyc,

We are sorry to bother you again. As the discussion period closes in two days, we would like to check whether our response and revision have resolved your concerns. We would greatly appreciate any feedback, and we remain happy to answer further questions.

Best regards, The Authors

Reviewer_SDf9OpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines how memory is added to a vision-language-action model through a gate. It shows that the gate value can be misleading: a small gate may be offset by a large memory projection, while a saturated gate leaves the projection to set the update size. BIND-VLA separates the direction and size of the memory update so that the gate controls a bounded update. The paper compares several fusion designs on MIKASA-Robo and checks whether adding memory hurts performance on LIBERO.

Critical strengths
  1. The paper identifies a clear problem and offers a solution that directly addresses it. The gate and memory projection can compensate for each other, so the gate alone does not tell us how much memory is being added. The proposed update removes this ambiguity and gives the gate a precise meaning. I found this connection between the diagnosis and the method convincing.

  2. The experiments do more than compare final success rates. Under the same memory setup, BIND-VLA improves over ordinary MLP gating and over normalization without orthogonalization. Fixing the learned update size to its mean at inference also reduces success from 38.4% to 29.2%. These comparisons provide useful evidence that the design choices matter. Although the individual tools are familiar, I think their use here offers a worthwhile contribution.

Critical weaknesses
  1. Scale coupling explains how the gate and projection can compensate for each other, but does not fully explain why training drives them to extremes. The designs use different projection initializations and almost no weight decay. I would like to know whether the same behavior holds with other reasonable training choices.

  2. Fixing the gate only at inference is not a substitute for training a fixed-gate baseline. Also, the LIBERO and MIKASA models are trained separately, so the results do not show that one policy can switch memory on and off across different task needs. These limit the interpretation, though I still find the main contribution worthwhile.

Questions for authors
  1. Do gate saturation and collapse persist with different initializations or stronger regularization of the memory projection?

  2. How does BIND-VLA compare with the same normalized, orthogonal update trained with a fixed gate value or a single globally learned gate value?

7 comments

We thank the reviewer for the precise and constructive comments. Both weaknesses are addressed with new experiments, all included in the revised PDF (changes in blue).

New since submission: (1) MLP retrained with the initialization of BIND-VLA, (2) MLP retrained with strong weight decay on the projection, (3) BIND-VLA retrained with a single globally learned gate, (4) three training seeds for the main configurations, and (5) probes of the original MemoryVLA and Engram designs.

W1 & Q1. Why training drives the gate to extremes, and whether this holds under other training choices.

We agree. Scale coupling only shows that training constrains the product of the gate and the projection scale, not how this product is split. We have added the missing mechanism (Appendix D.2) and tested initialization, regularization and training seeds. The collapse persists under all of them.

Mechanism. For the gate to end at an extreme, something must push it toward 0 or 1, and nothing must pull it back once it is there. Both hold in our designs, for different reasons.

  • Dot (saturation). The logit scales with D\sqrt{D} (Eq. 22), so a moderate alignment of qq and kk already saturates the gate, and the factor 1−g1-g in Eq. 21 then removes almost all gradient that could pull it back.
  • MLP (collapse). The projection scale grows during training, where the near-zero weight decay does not hold it back, and the gate shrinks so that the update keeps the size the loss prefers. Lowering a single bias shrinks the gate on every input at once (Eqs. 23–24). The update and hence the loss stay nearly unchanged along this path, so nothing pulls the gate back up.

Initialization. As the reviewer notes, our designs differ in projection initialization. We retrained MLP with exactly the initialization of BIND-VLA and probed it as in Figure 1.

ModelInitializationMean gatessuuSuccess (%)
MLPDefault (b2=0b_2 = 0, WO=0W_O = 0)0.00677.50.06228.1
MLPAs in BIND-VLA2.6×10−42.6 \times 10^{-4}38.50.01017.6
BIND-VLAb2=−3b_2 = -3, truncated-normal WOW_O0.08111.80.05738.4

The MLP gate collapses further while the projection grows to compensate, and success drops from 28.1% to 17.6%. BIND-VLA starts from the same initialization and keeps an active gate whose value sets the update (u=ρgu = \rho g). Initialization thus changes how far the MLP gate collapses, but not whether it collapses.

Regularization. Our weight decay of 10−1010^{-10} follows the π0.5\pi_{0.5} fine-tuning setup. To test whether holding the projection back restores control, we applied a decoupled weight decay of 10 to WOW_O alone, an extreme value that fully suppresses the compensation.

Weight decay on WOW_OMean gatessuu
10−1010^{-10} (default)0.00677.50.062
104.2×10−54.2 \times 10^{-5}0.027∼10−6\sim10^{-6}

The compensation is gone, but the gate still collapses, and the update now vanishes with it. Since g<1g < 1, a projection held small by weight decay caps the update at ss, so the gate cannot take over the magnitude. Regularization thus removes the memory instead of restoring gate control.

Dot gate. We did not retrain residual Dot under these settings. Two observations suggest its saturation does not hinge on them. It occurs within about 100 steps (Figure 3), and the same gate also saturates at 0.99 with a truncated-normal WOW_O, with and without a fixed offset of −3-3 on its logit (Dot + Ortho, Appendix I.2), although under decoupled rather than residual fusion.

Training seeds and released checkpoints. The pattern recurs across three training seeds: the mean Dot gate is 0.999 every time, and the mean MLP gate is 0.0067 / 0.0065 / 0.0072 (Appendix J). It also appears in the released MemoryVLA checkpoint, trained by its authors with a different backbone and recipe, where the gate hardly varies across tokens (std ≤ 0.011) and the update tracks the projection scale (Pearson ≥ 0.92, Appendix K).

Not inevitable. These driving forces depend on the training setup and are not implied by scale coupling alone. In the original Engram, a small language model trained from scratch, the gate keeps control (Pearson 0.95 between uu and gg, Appendix K). We therefore restrict the diagnosis to memory-augmented VLAs.

BIND-VLA is robust to these choices by construction. Its projection also grows during training (Figure 3b), but under decoupled fusion the update is invariant to the scale of WOW_O (Proposition 3), so neither this growth nor weight decay can change it. Its gate can still reach an extreme if the loss requires it, as on LIBERO, but then a small gate means a small injection that the projection cannot offset.

Revision. The mechanism is added to Appendix D.2, and the initialization and regularization experiments to the new Appendix I.2 (Table 16).

W2 & Q2. A trained fixed-gate baseline, and switching memory across task needs.

We agree with both points. We trained the requested baseline, and we have corrected the statements about switching.

Trained global gate. We retrained BIND-VLA with a single globally learned gate, with the same normalized, orthogonal update, budget and initialization. W2W_2 of the gate MLP is kept at zero, so only its bias is learned.

ModelAvg. success (%)
BIND-VLA38.4
Global learned gate (trained)33.2
BIND-VLA, Flattened at inference29.2
Dot32.4

Without input dependence, the trained model falls 5.2 points below BIND-VLA, to the level of Dot. Input-dependent modulation thus accounts for most of the 6.0-point gain over Dot. This run uses a single seed, and the gap is well above the paired standard deviations across training seeds (0.8–1.4 points, Appendix J).

The reviewer's concern is confirmed by the numbers. Flattening at inference costs 9.2 points, whereas training without input dependence costs 5.2. The inference-time intervention overstates the effect, and we now treat the trained model as the baseline.

Fixed gate value. We did not train hand-picked constants. The learned scalar lets training select the constant, so a fixed value is a special case of it, and we do not expect it to do better. One existing reference point is Dot + Ortho, which uses the same decoupled fusion with a gate saturated at about 0.99. This effectively fixes the update near the full budget, and it reaches only 11.3%.

Switching across task needs. We agree that separately trained models do not show one policy switching memory on and off. We have revised the corresponding statements in the abstract, the contributions, Sec. 5.3, Appendix G and the Conclusion. They now say that the same fusion rule learns to switch memory off when trained on tasks that do not need history. Within the single model trained jointly on the nine Remember tasks, the gate does vary across tasks, episode phases and tokens (Appendix G), but this is modulation rather than on/off switching.

Revision. The global learned gate is described in the new Appendix I.3 (Table 17), with a pointer in Sec. 5.3.

We thank the reviewer again for the constructive feedback. Both suggestions led to experiments that sharpened what the paper claims. We hope they resolve the remaining concerns, and we would be glad to discuss any further questions.

Thanks for the thorough rebuttal! The additional experiments address my concerns and strengthen the paper. However, I believe my original rating of 3 already reflects my positive assessment, so I will keep it unchanged.

Reviewer_om7eOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper asks whether the gate in a memory-augmented VLA actually controls memory injection. It points out that scale coupling between the gate and the projection can disconnect gate values from the actual injection strength. The proposed BIND-VLA uses orthogonalization and normalization to turn the projected memory into a unit direction, with a bounded gate controlling the relative update. It adds memory to a pretrained VLA through an external memory bank and per-token retrieval.

Critical strengths

S1. The diagnosis of Dot gate saturation is well supported and offers useful guidance for memory fusion design. Within the authors' shared architecture, the gates in Dot and Gated are close to saturation, and the relative update changes with projection scale. The training trajectories and within-episode statistics provide consistent evidence for this behavior. Looking at the gate, projection scale, and relative update together helps identify injection issues that are hard to assess from gate values alone.

S2. The fusion rule is simple, the gate adds little parameter overhead, and the key derivations hold. In non-degenerate cases where the clamp is not triggered, holds and the rotation angle is , so the gate directly reflects the local injection strength. The fusion rule consists of orthogonalization, normalization, and an MLP gate with about 1.0M parameters. It is straightforward to implement and diagnose.

S3. The reporting is transparent, with a good level of detail for reproducing the results. The appendices give training hyperparameters, initialization for each configuration, stop-gradient settings, numerical handling, and probe protocols. They also state the baselines' training budgets and where the checkpoint used in Figure 3 came from. These details make it easier to check the experimental conditions and reproduce the results.

Critical weaknesses

W1. It is still unclear how much of the performance gain comes from better gate control over injection strength. MLP + Unit already satisfies , but its average success rate does not exceed that of Dot or Gated , even though their gates are saturated. BIND-VLA differs from MLP + Unit only in orthogonalization. This comparison supports the value of orthogonalization, but it doesn't distinguish improved gate control over the effective update from other effects of orthogonalization on optimization. The initialization differences between MLP and BIND-VLA also make it harder to tell what drives the results.

W2. More statistical evidence is needed to show how stable the main performance gaps are. The main success rate results come from a single training seed. The paper doesn't report the distribution of results across repeated training runs or confidence intervals. Fixed evaluation seeds don't measure variation from training, so we still don't know how stable the main gains and ablation rankings are.

W3. The choice of the default and its effect on the evaluation results need to be clearer. gives the highest average success rate among six budget settings tested on the same nine evaluation tasks. The paper doesn't report a separate validation split or say whether the baselines were tuned to the same extent. Using evaluation results to support the default budget choice could introduce an optimistic bias into the reported gains.

Questions for authors

Q1. Can the same loss of gate control be reproduced in the original memory fusion designs? The current diagnosis comes from gate variants implemented within a shared memory architecture. Please clarify how these variants differ from the original MemoryVLA and Engram designs, and whether the same loss of gate control and scale compensation also occur in the original fusion forms. This would help establish how broadly the diagnosis applies.

Q2. Do the diagnostic findings and performance results depend on ? The experiments use only one VLA backbone and are all run in simulation. Could you check the key gate statistics and performance comparisons on a second backbone, or clarify which conclusions currently apply only to the tested setup?

Q3. What explains the drop on InterceptMedium, and what are the method's limits when handling dynamic information and longer histories? The Remember tasks mainly test short-term recall. On InterceptMedium, which requires estimating motion from history, BIND-VLA has a success rate of 33%, below the 43% achieved by without memory. Please analyze this gap and discuss the expected limitations on more complex memory-dependent manipulation tasks.

7 comments

We thank the reviewer for the careful reading and the specific questions. Each point below is answered with new experiments or analyses, all included in the revised PDF (changes in blue).

New since submission: (1) three training seeds for the main configurations, (2) BIND-VLA retrained with a global gate, (3) MLP retrained with the initialization of BIND-VLA, (4) a rotation analysis of MLP + Unit, and (5) probes of the original MemoryVLA and Engram designs.

W1. Does the gain come from better gate control over the effective update?

The new evidence indicates that it does. Orthogonalization helps only together with an input-dependent gate, and initialization does not account for the gain. Three analyses support this (new Appendix I, Tables 15–17).

(a) In MLP + Unit, the gate is identifiable but not yet in control. u=ρgu=\rho g holds, but the part of z^\hat z along hh only rescales hh and is largely removed by the next RMSNorm. The effective change is the rotation φ\varphi, which depends on both gg and ∠(z,h)\angle(z,h) (Eq. 31). In the trained model, cos⁡(h,z)\cos(h,z) grows with the gate (Pearson 0.87), so larger gates spend more of their update on rescaling.

Gate quintile1st2nd3rd4th5th
Mean gg0.0310.0420.0510.0620.071
Mean φ\varphi (°)1.151.581.932.212.14
Fraction of arctan⁡(ρg)\arctan(\rho g)93%94%94%89%75%

The top quintile has a 15% larger gate than the fourth yet rotates hh less, and tokens with similar gates there rotate by widely different amounts (P10–P90: 1.07–2.77°, more than twice the range of any other quintile). The projection scale took over the injection strength in Sec. 3. Here the projection direction co-determines the effective update, most strongly at the largest gates. This is consistent with MLP + Unit not exceeding Dot despite u=ρgu=\rho g. In BIND-VLA, φ=arctan⁡(ρg)\varphi=\arctan(\rho g) for every token.

(b) Initialization does not explain the gain. MLP + Unit and BIND-VLA share exactly the same initialization (b2=−3b_2=-3, truncated-normal WOW_O; Table 8), so the 6.6-point gap between them is unaffected. The reviewer is right that MLP differs (b2=0b_2=0, WO=0W_O=0). We therefore retrained MLP with the initialization of BIND-VLA. Its gate collapses further (from 0.0067 to 2.6×10−42.6\times10^{-4}), the projection compensates (s≈38.5s\approx38.5, u≈0.010u\approx0.010), and success drops from 28.1% to 17.6%. Likewise, matching the initial update of Dot + Ortho to that of BIND-VLA leaves it at 11.2% (11.3% before). The degeneration thus stems from the fusion rule and the gate, not from initialization.

(c) Orthogonalization helps only through the input-dependent gate. We retrained BIND-VLA with a single learned global gate shared by all inputs and crossed the two factors.

Input-independent gateInput-dependent gate
w/o orthogonalization32.4 (Dot$^\dagger$)31.8 (MLP + Unit)
w/ orthogonalization33.2 (global learned gate)38.4 (BIND-VLA)

†^\daggerThe Dot gate exceeds 0.99 on 99.9% of tokens and is thus effectively constant. It uses residual rather than Unit fusion, so this cell is a reference, not a strict factorial cell.

Orthogonalization with a global gate (+0.8 over Dot) and an input-dependent gate without orthogonalization (−0.6) both stay at the level of Dot. Only their combination yields the gain (+6.0). If orthogonalization improved optimization by itself, the global-gate model should benefit as well, and it does not. With the saturated Dot gate, orthogonalization even drops to 11.3% (Table 2). Flattening the gate at inference points the same way, as the gains of BIND-VLA over Dot and MLP + Unit shrink from 6.0 / 6.6 to −1.4 / 0.5 points.

The runs in (b) and (c) use a single seed. The 5.2-point gap between BIND-VLA and its global-gate variant is well above the paired standard deviations across seeds (0.8–1.4 points, see W2). We cannot exclude every other effect of orthogonalization, but we find no evidence of a benefit that is independent of the gate.

Revision. New Appendix I (Tables 15–17), with a pointer in Sec. 5.2. Appendix C now states that MLP and BIND-VLA differ in initialization as well as in the fusion rule.

W2. Statistical stability of the main results.

We agree that fixed evaluation seeds control only evaluation noise. We retrained the five main configurations with two additional training seeds, keeping all other settings identical (seed 1 is the run in the paper). BIND-VLA is the best design under every seed, and every paired 95% interval of its gap excludes zero.

ConfigurationSeed 1Seed 2Seed 3Mean ± StdGap to BIND-VLA95% CI of gap
No memory15.412.118.215.2 ± 3.123.2 ± 1.4[19.8, 26.6]
Dot32.429.834.732.3 ± 2.56.2 ± 0.8[4.3, 8.1]
MLP28.124.730.627.8 ± 3.010.7 ± 1.3[7.5, 13.9]
MLP + Unit31.828.834.531.7 ± 2.96.8 ± 1.2[3.9, 9.6]
BIND-VLA38.436.840.238.5 ± 1.7––

Std is the sample standard deviation over three seeds. Gaps are computed seed by seed, and the interval is a paired tt-interval (n=3n=3).

Within a seed, all configurations share the same data order, and all runs are evaluated on the same episodes, so configurations can be compared in pairs. Each configuration varies by 1.7–3.1 points across seeds, but all rise and fall together, so the paired gaps are much more stable (std 0.8–1.4 points).

The diagnosis is also stable. Across the three seeds, the mean Dot gate is 0.999 every time, the mean MLP gate is 0.0067 / 0.0065 / 0.0072, and that of BIND-VLA is 0.081 / 0.093 / 0.078.

Ablations. Repeating all ablations exceeded our compute budget for the rebuttal period, so they remain single-seed. Each ablated configuration shares its seed with the default one and is thus paired with it. Adjacent gaps in Table 4 are at least 4.8 points, more than three times the largest paired standard deviation above. Smaller differences, such as among Dot, Scaled Dot and Gated hh (within 1.5 points), are not treated as rankings, and none of our conclusions depends on their order.

Revision. New Appendix J (Table 18), with a pointer in Sec. 5.2.

W3. Choice of the default budget ρ\rho.

We agree. ρ=0.7\rho = 0.7 was selected among six budgets on the nine Remember tasks themselves, without a validation split, so the reported gain may be optimistic. We now state this in Sec. 5.4. Four checks indicate that the effect is small relative to the gain.

1. The gain does not rely on the best ρ\rho (Table 14).

ρ\rho0.10.20.30.50.70.99
Avg. success (%)31.632.134.936.038.426.7

With the runner-up ρ=0.5\rho = 0.5, BIND-VLA still exceeds Dot by 3.6 points and stays above every gate design, including the strongest, Gated hh (33.9%). Performance changes gradually between ρ=0.3\rho=0.3 and 0.70.7 and degrades only at the extreme ρ=0.99\rho = 0.99.

2. The choice is not driven by single tasks. Selecting ρ\rho on any eight of the nine tasks yields ρ=0.7\rho = 0.7 in all nine cases.

3. The gain does not regress on new training runs. ρ\rho was selected on seed 1, and seeds 2 and 3 in W2 were trained after it was fixed. BIND-VLA exceeds Dot by 7.0 and 5.5 points there, against 6.0 on seed 1. These runs share the evaluation episodes, so this check covers selection on training variation only.

4. ρ\rho was not retuned elsewhere. With the same ρ=0.7\rho = 0.7, BIND-VLA improves on ShellGameTouch (92% vs. 72% for Dot) and matches the memory-free policy on LIBERO (92.4% vs. 92.2%, Dot 84.8%). It is lower on InterceptMedium (33% vs. 39%), which we analyze in Q3.

Tuning of baselines. All methods share the training hyperparameters of Table 6, which were not tuned for BIND-VLA. Dot and MLP follow the gates of Engram and MemoryVLA, and their residual fusion has no counterpart of ρ\rho. No baseline hyperparameter was tuned, and ρ\rho is the only one selected for BIND-VLA.

Revision. Sec. 5.4 and Appendix H.2.

Q1. Does the loss of gate control occur in the original fusion designs?

Yes for the original MemoryVLA, and no for the original Engram in a language model. We have narrowed our claim accordingly.

Our variants share one memory architecture and keep only the gate of each original design, so that the gates can be compared alone.

EngramDotMemoryVLAMLP
MemoryNN-gram lookupRetrieved historyTwo memory banksRetrieved history
GateDot product, signed sqrtDot productElement-wise MLPScalar MLP
FusionResidual, short conv.ResidualConvex combinationResidual

We measured the quantities of Figure 1 on both original designs.

Original MemoryVLA. It fuses memory as x~=g⊙x+(1−g)⊙H\tilde x = g\odot x + (1-g)\odot H, i.e. the residual update x+(1−g)⊙(H−x)x + (1-g)\odot(H-x). We define u=∥(1−g)⊙(H−x)∥/∥x∥u=\|(1-g)\odot(H-x)\|/\|x\| and s=∥H∥/∥x∥s=\|H\|/\|x\|, and probe its released MIKASA checkpoint on RememberColor. The gate hardly changes from token to token (std ≤ 0.011 in both memory streams). It varies mainly across channels and thus acts as a fixed set of channel weights rather than an input-dependent injection strength. The injection strength is set by the projection instead. uu closely tracks ss (Pearson ≥ 0.92), and in a joint regression, a one-std change in ss changes uu 5 to 50 times more than a one-std change in gg. This is the same loss of control as for Dot and Scaled Dot in Sec. 3.

Original Engram. Engram targets language models, so we build on its official demo implementation and train a 22.5M-parameter model on WikiText-103. Here the gate keeps control. uu follows gg (Pearson 0.95) and ss stays narrow (0.48±0.060.48 \pm 0.06), also without weight decay (Pearson 0.97). Scale coupling therefore permits a loss of control but does not force it, since drifting along the coupling (Eq. 3) needs a driving force. We conjecture that this force is weaker in this setting, although we have not isolated the individual factors. One candidate is the small model size, which bounds the dot-product logit (Eq. 22). Another is the training regime. The model is trained from scratch with supervision at every token, whereas a VLA is fine-tuned from pretrained representations and its action loss likely reaches each token only indirectly.

Scope. The loss of control occurs in all our residual variants and in the original MemoryVLA, but it is not inevitable. We now restrict the diagnosis to memory-augmented VLAs and no longer extend it to gated modules in general (Conclusion). BIND-VLA does not depend on such training dynamics, since decoupled fusion sets the update by the gate alone by construction.

Revision. New Appendix K (Table 19), with a pointer in Sec. 3.

Q2. Dependence on the backbone and the simulated setting.

The diagnosis holds on a second VLA backbone. The performance results are so far limited to π0.5\pi_{0.5} in simulation, as we have not trained BIND-VLA on a second backbone, and we now state this scope explicitly.

Gate statistics on a second backbone. The MemoryVLA checkpoint from Q1 builds on the 7B Prismatic VLM pretrained on Open X-Embodiment (DINOv2 and SigLIP encoders, a 7B LLaMA language model, a DiT action expert) rather than on π0.5\pi_{0.5}.

BackboneFusionGate across tokensInjection set by
π0.5\pi_{0.5}Dot>0.99>0.99 on 99.9%Projection
π0.5\pi_{0.5}MLPmean 0.0067Projection (s=7.5s=7.5)
Prismatic 7B (MemoryVLA)Cognitive0.888±0.0070.888 \pm 0.007Projection (rs,u=0.92r_{s,u}=0.92)
Prismatic 7B (MemoryVLA)Perceptual0.528±0.0110.528 \pm 0.011Projection (rs,u=0.98r_{s,u}=0.98)
π0.5\pi_{0.5}BIND-VLA0.081, CV ≈ 50%Gate (u=ρgu=\rho g)

Despite a different backbone, memory and fusion form, MemoryVLA loses gate control just as our residual variants do.

Which conclusions apply where.

  • Any backbone, by construction: the properties of Eqs. 7–8, namely u=ρgu=\rho g, the rotation arctan⁡(ρg)\arctan(\rho g) and the bound ρ\rho.
  • Two VLA backbones, empirically: the loss of gate control, which may still depend on the training setup, as Engram shows in Q1.
  • π0.5\pi_{0.5} in simulation only: the performance gains, ρ=0.7\rho=0.7 and injection at layer 1, which may need re-tuning elsewhere, including on real robots.

Revision. Table 20 in the new Appendix K. The scope is stated there and in the Conclusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.