- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies how positional interference in the memory of VLA can change its behavior. The paper changes the positional information on 3 tasks in the RoboMME benchmark and tests the changes in behavior. The paper introduces a controlled intervention that permutes memory while either reassigning positional indices based on new locations or preserving their original positional identities. Experiments on FrameSamp-Modul and TokenDrop-Modul show that reassigning positions changes predicted actions, whereas preserving original positions exactly recovers the reference predictions. However, a separate closed-loop evaluation does not demonstrate improved task success.
Critical strengths
- The paper aims to study an important problem of VLA memory and interfere with the memory in a controlled and effective way. The paper verifies that changing the positional information indeed alters the VLA's behavior.
Critical weaknesses
-
The observations in the paper do not contribute to a clear explanation of the memory mechanism of VLA. The experimental results only show that the VLA behavior is sensitive to positional information. However, they do not verify a solid mechanism or reveal any intriguing phenomenon that can explain the interference or trigger future research/designs. The exact recovery under W is not suprising either, as it generally follows from the properties of RoPE and attention.
-
The experimental setting is quite restricted. It is only tested on three tasks of RoboMME and only one memory-based VLA model is tested. The experiments mostly use only one seed and the interference is based on one permutation method. The number of trials are also limited (30).
-
The behavioral experiment does not validate the central mechanism. The closed-loop test changes upstream temporal embeddings, while the principal localization concerns downstream serialized key positions.
Questions for authors
Please address the weakness part.
We agree that W recovering U is expected from attention invariance. With a fixed query, jointly permuting keys, values, and their mask entries permutes the corresponding attention weights and preserves their weighted sum in exact arithmetic. Transporting each key’s original position also preserves its RoPE rotation. We use W recovery as a control on the intervention and numerical execution.
The new empirical result concerns the original divergence that persists after retention matching. Holding each history’s upstream temporal assignment fixed, sharing the FAR downstream key-position map reduced first-16-step divergence by 0.002948 MAE on average across 27 episode configurations evaluated under three inference seeds (95% task-stratified episode-bootstrap CI [0.001999, 0.004049]). The reduction exceeded both tested shuffle controls, and the average effect reversed in VideoPlaceOrder. Attention invariance explains recovery under joint transport. The effect of changing the assignment on the original divergence is measured by this new experiment.
We independently recomputed the archived TokenDrop results from the saved action arrays. Across 27 distinct episodes and three process repeats, transporting positions recovered canonical U bitwise and reassigning positions displaced the actions in every evaluation. The median first-16-step MAE between reassigned and canonical reference actions was 0.002191. This supports positional sensitivity across two selectors sharing the same downstream modulation interface. The causal bridge was tested in FrameSamp, and the inference seeds repeat evaluation on the same episode configurations. Generalization to an independent architecture remains open.
We also tested a downstream intervention in closed loop, addressing the distinction from the earlier upstream temporal-embedding experiment. The online age rule yielded zero rescues and zero losses across 15 paired development configurations. A second position rule changed predicted actions on a matched replay but left the diagnosed wrong-grasp failure in place in closed loop. We will include these negative results to clarify the behavioral limits of the tested interventions.