acceptodds
Under review as a conference paper at ICLR 2027

Beyond Retention: Causal Localization of Positional Interference in Memory-Augmented Vision-Language-Action Models

Abstract

Memory-augmented vision-language-action (VLA) policies can become sensitive to long and distracting histories even when task-relevant evidence remains available. We ask whether such sensitivity arises solely from retention or also from how retained memory is positionally represented downstream. On RoboMME, we construct controlled FAR/MIDDLE/RECENT histories and retention-matched counterfactuals for a memory-augmented VLA. Across 30 episodes from three task families, moving semantically matched historical evidence changes the initial predicted action chunk, and forcing the exact same retained frames does not generally eliminate this divergence. We then localize two positional interfaces in the tested memory-as-modulation architecture: per-frame temporal encoding before memory attention and serialized memory-token key positions within RoPE-based memory attention. To isolate the latter, we permute represented memory while either assigning new serialized key positions or transporting each memory unit's original key positions under matched reduction order. In FrameSamp-Modul, position transport recovers the reference action output bitwise for all 27 primary episodes in each of three fresh process blocks, whereas assigning new serialized key positions produces displacement beyond the predeclared nuisance criterion. We further reproduce this positional-transport effect in TokenDrop-Modul, which changes the upstream perceptual-memory representation while retaining the downstream modulation interface. Finally, a prospective 120-rollout closed-loop evaluation does not confirm the hypothesized behavioral improvement from recent history or temporal-position correction. Together, these results separate memory availability, positional representation, action sensitivity, and closed-loop behavior, and show that a reproducible action-level positional mechanism does not by itself imply a confirmed task-success benefit.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, embodied AI, robot learning, memory-augmented agents, memory interference, positional encoding, mechanistic analysis, long-horizon manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XQj3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how positional interference in the memory of VLA can change its behavior. The paper changes the positional information on 3 tasks in the RoboMME benchmark and tests the changes in behavior. The paper introduces a controlled intervention that permutes memory while either reassigning positional indices based on new locations or preserving their original positional identities. Experiments on FrameSamp-Modul and TokenDrop-Modul show that reassigning positions changes predicted actions, whereas preserving original positions exactly recovers the reference predictions. However, a separate closed-loop evaluation does not demonstrate improved task success.

Critical strengths
  1. The paper aims to study an important problem of VLA memory and interfere with the memory in a controlled and effective way. The paper verifies that changing the positional information indeed alters the VLA's behavior.
Critical weaknesses
  1. The observations in the paper do not contribute to a clear explanation of the memory mechanism of VLA. The experimental results only show that the VLA behavior is sensitive to positional information. However, they do not verify a solid mechanism or reveal any intriguing phenomenon that can explain the interference or trigger future research/designs. The exact recovery under W is not suprising either, as it generally follows from the properties of RoPE and attention.

  2. The experimental setting is quite restricted. It is only tested on three tasks of RoboMME and only one memory-based VLA model is tested. The experiments mostly use only one seed and the interference is based on one permutation method. The number of trials are also limited (30).

  3. The behavioral experiment does not validate the central mechanism. The closed-loop test changes upstream temporal embeddings, while the principal localization concerns downstream serialized key positions.

Questions for authors

Please address the weakness part.

1 comment

We agree that W recovering U is expected from attention invariance. With a fixed query, jointly permuting keys, values, and their mask entries permutes the corresponding attention weights and preserves their weighted sum in exact arithmetic. Transporting each key’s original position also preserves its RoPE rotation. We use W recovery as a control on the intervention and numerical execution.

The new empirical result concerns the original divergence that persists after retention matching. Holding each history’s upstream temporal assignment fixed, sharing the FAR downstream key-position map reduced first-16-step divergence by 0.002948 MAE on average across 27 episode configurations evaluated under three inference seeds (95% task-stratified episode-bootstrap CI [0.001999, 0.004049]). The reduction exceeded both tested shuffle controls, and the average effect reversed in VideoPlaceOrder. Attention invariance explains recovery under joint transport. The effect of changing the assignment on the original divergence is measured by this new experiment.

We independently recomputed the archived TokenDrop results from the saved action arrays. Across 27 distinct episodes and three process repeats, transporting positions recovered canonical U bitwise and reassigning positions displaced the actions in every evaluation. The median first-16-step MAE between reassigned and canonical reference actions was 0.002191. This supports positional sensitivity across two selectors sharing the same downstream modulation interface. The causal bridge was tested in FrameSamp, and the inference seeds repeat evaluation on the same episode configurations. Generalization to an independent architecture remains open.

We also tested a downstream intervention in closed loop, addressing the distinction from the earlier upstream temporal-embedding experiment. The online age rule yielded zero rescues and zero losses across 15 paired development configurations. A second position rule changed predicted actions on a matched replay but left the diagnosed wrong-grasp failure in place in closed loop. We will include these negative results to clarify the behavioral limits of the tested interventions.

Reviewer_obd4OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether historical-position sensitivity in memory-augmented VLA policies persists after retained frames are matched. It compares original represented memory and key positions (UU), permuted memory with reassigned positions (VV), and the same permutation with original positions transported (WW). Under canonical matched numerical reduction order, WcW_c recovers UcU_c bitwise on 27 primary RoboMME episodes across three process runs, while VcV_c remains displaced. The subscript cc denotes the canonical computation. The finding reproduces with TokenDrop memory sharing the downstream modulation interface. A separate 120-rollout experiment does not confirm improvement from an upstream temporal-embedding correction.

Critical strengths
  1. The comparison separates represented content from positional assignment. Conditions VV and WW contain the same permuted memory, so the final action difference is more informative than an uncontrolled history shuffle. Recovery and displacement are reported across both FrameSamp and TokenDrop, supporting the finding within their shared downstream interface.

  2. The paper reports its limitations carefully. It distinguishes scientific episodes from process repeats, preserves the original 1/31/3 strict-recovery result, and reports the negative closed-loop outcome. It also explicitly avoids claiming replication across independent architectures or a demonstrated behavioral repair.

Critical weaknesses
  1. The explanatory contribution remains narrow. Section 5.5 acknowledges that joint permutation of memory, positions, and masks preserves attention in exact arithmetic. Reassigning RoPE positions changes the attention computation, so the observed action dependence does not itself distinguish harmful interference from useful temporal sensitivity. The direct bridge in Section 4.3 is also unconfirmed: the median projection is -0.031, with positive projections in only 48.3% of directed comparisons. The downstream intervention establishes a separate local dependence, but does not explain the original residual history effect.

  2. The central controls are insufficiently specified for independent assessment. Sections 5.3, D.3, and H.3 do not explain where tensors are reordered relative to key rotation, masking, softmax, and value aggregation, or how canonical UcU_c compares numerically with native UU. Appendix E supplies displacement bounds without the underlying legacy-control measurements. Separately, Figure 5 defines directed interventions with source-specific content assemblies, but their relationship to the natural and retention-matched distances in Table 1 remains unclear. Appendices B and C do not give a complete episode construction.

Questions for authors
  1. Can holding serialized key-position assignment fixed reduce the residual divergence in the existing retention-matched comparisons? A limited intervention connecting these experiments would help distinguish an explanation of interference from a separate positional dependency.

  2. How exactly is reduction order matched? Please provide the canonical procedure, deviations between native UU and canonical UcU_c, and the numerical-control measurements underlying the displacement gates.

  3. Which action arrays enter the directed comparisons in Figure 5 and the natural/matched comparisons in Table 1? Please give one complete history construction and the shared-frame selection rule, and provide access to the existing manifests and result arrays.

1 comment

Thank you for identifying the missing link between the original retention-matched interference and the downstream position intervention. We tested this link directly by assigning corresponding retained memory keys the same FAR position map while preserving each history’s own upstream temporal assignment. The baseline and intervention used matched canonical reduction order. In a trained-checkpoint diagnostic, the intervention changed 432 of 512 key-position IDs and the resulting key rotations while preserving the unrotated memory keys and initial query tensors.

Across 27 distinct episode configurations evaluated with three inference seeds, shared FAR positions reduced first-16-step action divergence from 0.005672 to 0.002724 MAE on average. The absolute reduction was 0.002948 MAE (95% task-stratified episode-bootstrap CI [0.001999, 0.004049]), about 52% of the pooled baseline. The paired reduction exceeded the common-shuffle control by 0.001300 and the mismatched-shuffle control by 0.002089. The average effect was positive in VideoUnmask and InsertPeg and reversed in VideoPlaceOrder. This supports a partial downstream causal contribution to the retention-matched divergence.

Canonical reduction order. The model applies RoPE using the assigned key-position IDs, then gathers the already-rotated keys, values, and mask together into a common retained-token order. Attention logits, masking, softmax, and value aggregation follow that gather. The canonical gather therefore matches the order of the attention computation while preserving each key’s assigned rotation. In the new FrameSamp bridge, the pooled native-versus-canonical numerical diagnostic was 0.000093 MAE. In the archived TokenDrop replication, native and canonical U matched bitwise in all 81 episode–process evaluations. These measurements describe their respective experiments. The original FrameSamp displacement thresholds and underlying legacy-control measurements still require a separate audit trail.

Worked example from the new bridge. In the InsertPeg engineering episode, the FAR, MIDDLE, and RECENT conditions shared the same 32 semantic frame identities. Each frame contributed 16 tokens. The shared FAR map assigned each corresponding frame the slot it occupied in FAR and preserved each token’s within-frame offset. Relative to their native layouts, this changed 432 of 512 key-position IDs in MIDDLE and 496 in RECENT. For the FAR/MIDDLE pair, first-16-step MAE decreased from 0.029778 to 0.005370 under the shared FAR map. Across all three pairs in this episode, the common shuffle reduced divergence by almost the same amount as shared FAR. We therefore assessed specificity using paired shuffle comparisons across the full cohort.

Relation to the original Table 1 and Figure 5. Table 1 compares actions across natural and retention-matched histories. Figure 5 uses source-specific content assemblies for directed interventions and measures whether the induced action change projects toward the action difference between histories. We retain the original negative projection result (median −0.031, 48.3% positive). The worked example above documents the new intervention. The exact saved-array identifiers and a fully worked construction for the original Table 1 and Figure 5 comparisons are still needed to complete that audit trail.

Reviewer_zGxGOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the mechanistic role of serialized memory-key positions in memory-augmented VLA policies. Through controlled interventions, the authors demonstrate that reassigning key positions for fixed historical observations causally alters predicted actions, effectively isolating this sensitivity from the mere availability of task-relevant evidence. While this positional effect is rigorously localized within the modulation architecture, a separate closed-loop evaluation indicates that correcting these temporal positions does not necessarily translate to improved task success.

Critical strengths

N/A

Critical weaknesses

While the paper presents counterfactual mechanistic interpretation experiments, a major concern is the lack of clarity regarding their practical utility. It remains unclear how the identified mechanistic insights translate into actionable strategies for improving overall task performance.

Questions for authors

N/A

1 comment

Thank you for asking whether the localized positional mechanism can lead to better robot performance. We tested a downstream intervention that the policy can compute online from its current memory and frame timestamps. It changes memory-key positions at RoPE while retaining the original frame selector and upstream temporal encoding. A matched-input test on the trained checkpoint verified that the rule changed 192 of 512 key-position IDs while preserving the unrotated keys and initial query. Native and explicit-identity inference produced bitwise-identical actions on that input.

In closed-loop development evaluation on 15 paired RoboMME episode configurations, the age-based rule produced zero task-success rescues and zero losses against either the native policy or the identity-hook control. We also tested a second rule that assigns separate downstream position ranges to demonstration and execution frames. On a reproducible wrong-container failure, the rule changed the action chunk in exact replay, yet the benchmark still flagged a wrong-container grasp when the rule was applied at the final policy call or from the first eligible call.

These experiments demonstrate that the downstream intervention is feasible during closed-loop control. The tested rules showed no task-success gain in development. We will report these results separately from the earlier upstream temporal-embedding experiment and limit the practical claim to what the evaluations support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.