acceptodds
Under review as a conference paper at ICLR 2027

MemAlign: Memory–Action Alignment for Vision-Language-Action Models

14 views

Abstract

Vision-language-action (VLA) models have achieved remarkable progress in robotic manipulation, particularly when augmented with memory to support history-dependent decision-making under partial observability. However, memory-augmented policies may fail to translate historical evidence into appropriate actions, a gap we refer to as memory-action misalignment. In this work, we introduce MemAlign, a post-training framework that aligns action generation with historical evidence retained in memory, enabling policies to resolve ambiguous action choices using historical context. Specifically, we first introduce counterfactual supervision to train the policy to select appropriate actions for different histories under the same current observation and instruction. Then, we propose a history-aware low-rank modulation module to dynamically modulate the action network based on historical evidence, allowing different histories to induce different effective action mappings. Finally, we complement this modulation with an optional visual rebinding module that locates remembered targets in the current scene. Their estimated positions then guide the refinement of the generated actions. Extensive experiments on two long-horizon manipulation benchmarks (MIKASA-Robo and RoboMME) demonstrate that MemAlign significantly improves the success rate of completing history-specified targets, e.g., raising MemoryVLA’s result from to on MIKASA-Robo tasks. Notably, MemAlign exhibits strong generalization capabilities, consistently improving performance when applied to different memory-augmented VLA models.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied AI, Vision-Language-Action Models, Robotic manipulation, Memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RKCnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MemAlign, a post-training framework for improving memory–action alignment in memory-augmented VLA models. It combines counterfactual supervision with history-aware action modulation, and optionally uses visual rebinding for spatial grounding. Experiments on MIKASA-Robo and RoboMME show improved performance on memory-dependent manipulation tasks.

Critical strengths

1.The paper identifies a clear and meaningful problem. Memory–action misalignment is well motivated, and the original/replaced/erased history evaluation gives a useful way to test whether the policy is actually using memory to guide its actions.

2.The improvement on the controlled MIKASA setting is substantial. MemAlign significantly improves over the locally evaluated MemoryVLA baseline, showing that the proposed approach can effectively redirect actions based on historical information.

Critical weaknesses

1.The comparison with existing methods is not fully fair. The strongest MIKASA results use a controlled evaluation protocol that differs from the protocols used by the published baselines, so the claimed advantage over prior methods is hard to judge directly.

2.It is unclear how much of the gain comes from the core MemAlign design. For example, the RememberColor results also rely on a task-specific visual rebinding and metric grounding pipeline, while the main Shell result uses a different warm-start and recovery-data setup from the ablation study.

3.The evidence for general memory–action alignment is still limited. The strongest results mainly come from controlled target-selection tasks with fixed cue timing, while the gains on the broader RoboMME benchmark are relatively small and uneven across task categories.

4.There is no real-robot evaluation. All experiments are conducted in simulation, leaving the effectiveness and practicality of MemAlign in real-world deployment unclear.

Questions for authors

1.Why are the strongest memory baselines not re-evaluated under the same controlled MIKASA protocol as MemAlign? This would make the comparison with prior methods much more convincing.

2.The main Shell result reaches around 90%, while the full model in the matched ablation achieves only about 52–56% and uses a different training setup. Can the authors provide ablations under the same setup as the main Shell result?

3.How exactly is MME-VLA + MemAlign implemented and trained in Table 3? The paper reports an improvement from 44.51% to 45.62%, but the corresponding adaptation and training details are unclear.

7 comments
AuthorsOpenReview ↗

Response to Reviewer RKCn

We thank the reviewer for the insightful and constructive comments! In the following, we provide our point-by-point response and hope our response helps address your concerns. We also look forward to the subsequent discussion which may further help to solve the current issues.

W1 & Q1. Evaluation under the original and controlled protocols

We thank the reviewer for raising this point. We agree that the original manuscript did not clearly separate the two evaluation protocols. We have now evaluated both MemoryVLA and MemAlign under both protocols.

Why a controlled protocol. In MemoryVLA's original setting, the policy can already act while the cue is still visible. Actions planned during this period can carry over after the cue disappears, and the robot's cue-driven movement can itself encode the target. Success therefore mixes two effects: using memory, and continuing actions started while the cue was visible. Our controlled protocol adds a stationary cue-observation phase: the robot observes the cue without moving and only acts after the cue disappears. This isolates whether the target choice is driven by memory. Both policies are evaluated under the same protocol.

Table R3.1. Success rate (%) with and without the stationary cue-observation phase (mean ± std over three evaluation blocks).

ProtocolModelRC3RC5RC9ShellAverage
Controlled (with phase)MemoryVLA30.67 ± 3.062.00 ± 2.0010.67 ± 4.6216.67 ± 4.6215.00 ± 2.00
Controlled (with phase)MemAlign98.67 ± 1.1597.33 ± 2.3166.00 ± 5.2989.33 ± 2.3187.83 ± 2.52
Original (without phase)MemoryVLA38.00 ± 6.0020.00 ± 7.2114.00 ± 6.0086.67 ± 4.1639.67 ± 0.29
Original (without phase)MemAlign70.00 ± 5.2929.33 ± 4.1625.33 ± 3.0677.33 ± 9.4550.50 ± 3.77

Experiments under both controlled and original protocols. To make the comparison explicit, we evaluated MemoryVLA and MemAlign under both protocols (Table R3.1). Under the controlled protocol, MemAlign raises the average success rate from 15.00% to 87.83% and improves every task. Under the original protocol, MemAlign still improves the average from 39.67% to 50.50%, with consistent gains on all three RememberColor tasks.

The exception is ShellGameTouch under the original protocol, where MemoryVLA scores higher than MemAlign (86.67% vs. 77.33%). This result also helps explain why the protocols differ. MemoryVLA's 86.67% under the original protocol is consistent with its published 88%, yet the same model drops to 16.67% under the controlled protocol. This suggests that its original performance mainly comes from actions started while the cue is still visible, rather than from using memory after the cue disappears. The controlled protocol removes this shortcut and therefore isolates the memory-dependent target selection that MemAlign is designed to improve. This also accounts for the gap between the published MemoryVLA score in Table 1 and our controlled result in Fig. 3.

In the revision, we report both protocols side by side and clearly separate matched local comparisons from published baseline results obtained under their source protocols.

W2 & Q2. Ablations matched to the main experiment

We thank the reviewer for identifying this issue. The original ablation study and the main ShellGameTouch experiment started from different checkpoints. We acknowledge that this mismatch limited their direct comparability and prevented a fully controlled assessment of component contributions under the main configuration. To address this concern, we have rerun the ablation study using the same initial checkpoint, training data, training budget, and action-sampling settings as the main experiment. The updated results and detailed experimental setup are provided in the appendix. We have added this experiment to the manuscript in Line 471, Table 4.

Table R3.2. Matched component ablation on ShellGameTouch. Exclusive completion (%) ↑.

History-aware modulationCounterfactual supervisionVisual rebindingOriginalReplacedBoth histories
———21.33 ± 1.1522.67 ± 5.030
—✓✓38.00 ± 2.0040.67 ± 2.312.67 ± 1.15
✓—✓67.33 ± 2.3165.33 ± 6.4352.00 ± 5.29
✓✓✓90.00 ± 2.0092.00 ± 2.0084.67 ± 1.15

W3. Scope of the improvement and broader memory capabilities

We agree that MemAlign achieves its strongest improvements on the target-selection tasks in MIKASA-Robo, while its gains on RoboMME are more modest and uneven across task categories.

Why MIKASA-Robo is the primary setting. These tasks allow a direct test of memory–action alignment. We hold the current observation and instruction fixed and change only the historical cue. We can then check whether the policy selects the target that this history specifies. This is an interpretable test of exactly the behavior MemAlign is designed to improve.

Evidence beyond controlled cue timing. RoboMME does not use our controlled handover: cues appear at their native times and are followed by unconstrained robot motion. Even so, the gains follow the pattern we would expect.

  • They are largest on Reference tasks (+8.5 pp), where history determines which target to act on.
  • They are smaller on categories that depend on counting or imitation, such as Counting (−0.5 pp).
  • Average success improves for all three memory-augmented VLAs we evaluated (Table 3).

Counterfactual supervision currently covers all four Permanence tasks and three Reference tasks (PickHighlight, VideoPlaceButton, and VideoPlaceOrder). VideoRepick and all Counting and Imitation tasks are trained without the auxiliary counterfactual objective (see our response to Reviewer 1Cz8, W2).

Revised claims and limitations. We have revised the abstract and introduction to state that MemAlign primarily improves history-dependent target selection, rather than memory capabilities in general. App. G.2 now explicitly states that the MIKASA results rely on controlled cue timing and do not establish benefits across all memory types. In future work, we plan to construct counterfactual supervision for more diverse memory requirements and to evaluate transfer on further memory benchmarks.

Q3. Implementation and training details for MME-VLA

We thank the reviewer for this question. We have expanded App. D, and summarize the key details below.

Architecture. We use the released MME-VLA (perceptual Frame Sampling + Modulator) and keep all of its parameters frozen: vision, language, memory, and action expert. MemAlign adds two modules:

  • History controller. A two-layer, width-256 Transformer reads MME-VLA's native per-frame perceptual and memory tokens. The current frame is excluded, so the controller sees only past frames. It outputs coefficients based on how the action response under the actual history differs from that under a fully masked history. With no history, the coefficients are zero.
  • History-aware low-rank modulation. Rank-8 adapters are applied to the 18 memory-modulation projections and the 18 action FFN layers. Their strength is scaled by the controller's coefficients.

The adaptation adds 3.1M parameters. It does not use visual rebinding.

Counterfactual supervision. Each training pair shares the same current observation, robot state, and instruction, but has two different valid histories, each with its own expert continuation. The loss combines MME-VLA's native flow-matching loss with a counterfactual term. This term trains the policy to match each history's expert action and to change its action when the history changes. An erased-history teacher keeps behavior unchanged when no history is available.

Training. We train for 224 AdamW updates (learning rate 1e-4). Each update uses one counterfactual pair and two ordinary samples, weighted 0.25 / 0.75. Checkpoints are selected on development data only.

Evaluation and the updated numbers. The original submission compared our result (45.62%) with the published MME-VLA score (44.51%). In the revision, we rerun the released MME-VLA checkpoint locally under the same protocol as our adapted model: all 16 tasks, 50 episodes each. Under this matched comparison, MemAlign improves MME-VLA from 45.38% to 46.50% (Table 3).

W4. Real-robot evaluation

We thank the reviewer for emphasizing real-world validation. We have added a real-robot evaluation to the revised manuscript (Sec. 5.4, App. F).

Setup. We use a PiPER-X arm with a fixed RealSense D435 camera and a wrist-mounted D405 camera, both recording 640 × 480 RGB through ROS 2. The policy takes both camera views, proprioception, the instruction, and visual history as input, and predicts 16 future joint targets on a 10 Hz control grid. Training uses 108 demonstrations per task and has two stages:

  • Stage 1 adapts the native diffusion action expert, conditional LoRA, and the wrist/embodiment adapters for 20,000 updates, with an effective batch size of eight.
  • Stage 2 freezes the native action expert and wrist/embodiment adapters, and trains conditional LoRA, the history controller, and the coefficient generator for up to 3,000 updates with action fitting and counterfactual supervision.

Vision and memory remain frozen in both stages. Training uses four L40S GPUs with DDP; the native action expert is updated in Stage 1 and frozen in Stage 2 (App. F.3, Tables 20–21).

Tasks.

  • RememberColor3: the robot sees a transient color cue, then touches the block of the matching color among three candidates.
  • ShellGameTouch: the robot observes which cup hides the ball, waits, and then touches that cup.

We run 10 trials per task and history condition, and all methods use the same held-out layouts.

Table R3.3. Real-robot exclusive target completion (successes / 10 trials). The MemoryVLA baseline is adapted to the same embodiment with the same 108 demonstrations per task, without history-aware modulation or counterfactual supervision. Erased removes the cue and is scored against the original target; it is a diagnostic, not a success metric.

TaskMethodOriginal ↑Replaced ↑Erased
ShellGameTouchMemoryVLA4/102/103/10
ShellGameTouchMemAlign6/104/102/10
RememberColor3MemoryVLA1/102/101/10
RememberColor3MemAlign5/103/101/10

Results. On ShellGameTouch, MemAlign improves Original completion over MemoryVLA from 4/10 to 6/10. Under replaced histories, it redirects 4/10 trials to the new target, compared with 2/10 for MemoryVLA. Erasing the cue reduces MemAlign's original-target completion from 6/10 to 2/10, consistent with sensitivity to the cue history. On RememberColor3, MemAlign improves Original completion from 1/10 to 5/10 and Replaced completion from 2/10 to 3/10; its original-target completion under erased histories is 1/10. Failures generally fall into two categories: the gripper is not correctly oriented toward the target object, or it is oriented toward the target but fails to make proper contact. This can be observed in the videos provided in the supplementary materials.

Practicality. The 16-step profiling results in App. F.2, Table 19 show about 11% additional query latency and 0.8 GiB of peak GPU memory relative to MemoryVLA. The measured RC5/Shell query times are summarized in our response to Reviewer Zzha, W1 & Q2. On the real robot, RC3 query latency is 497.65 ms. The policy predicts 16 joint targets on a 10 Hz temporal grid, and RTC overlaps inference with buffered action execution. Local interpolation provides joint-target commands at 30 Hz, separating the command publication rate from the approximately 2 Hz compute-limited policy query rate.

We present these results as a feasibility study rather than a full real-world benchmark, and we note this in the limitations. Larger-scale real-world evaluation is an important direction for future work.

Reviewer_1Cz8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper targets memory–action misalignment in memory-augmented VLAs, the history contains enough evidence to identify the correct target, but the policy fails to act on it. The authors propose MemAlign, a post-training framework on a frozen MemoryVLA with three components: (a) counterfactual supervision, which pairs alternative histories under matched current inputs with their own expert actions and adds a centered action-contrast loss (b) history-aware low-rank modulation, in which memory-dependent coefficients gate a low rank adapter, and (c) optional visual rebinding, which localizes the remembered target in the current frame and refines the action. Under a controlled protocol, MemAlign raises target completion on four MIKASA-Robo tasks and It also gives modest gains on RoboMME and preserves LIBERO/SIMPLER performance.

Critical strengths

Strengths

  • Clear problem formulation; the paper is generally easy to follow.
  • The original/replaced/erased-history protocol is a well-designed diagnostic that directly tests whether the target choice follows memory.
Critical weaknesses

Weaknesses

[W1] Limited generalizability. Misalignment is quantified only for MemoryVLA, and Fig. 1 is schematic, so it is unclear whether other memory-augmented VLAs suffer from this problem to a comparable degree. Likewise, the full method is applied only to MemoryVLA. MemER receives only the counterfactual supervision, and the resulting gain is marginal (42.38% → 44.00%) and measured against a published number rather than a matched evaluation.

[W2] Unclear necessity of each component. The components used differ across settings. MemER uses neither modulation nor visual rebinding, rebinding is used only for RememberColor, and the RoboMME variant lists no counterfactual term in its objective (Table 6). This makes it hard to identify the contribution of each component, or what constitutes the MemAlign method, across different models and tasks.

Questions for authors

Questions

[Q1] Using the same protocol, can you quantify misalignment on other existing memory-augmented VLAs (e.g., MemER, MME-VLA, recurrent-memory VLAs) to show that the problem is general?

[Q2] Does the full MemAlign (all three components) transfer to at least one other memory-augmented VLA? If not, could you clarify when each component is required?

[Q3] What are the inference latency, additional memory, and training cost relative to MemoryVLA?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 1Cz8

We thank the reviewer for the insightful comments! In the following, we provide our point-by-point response and hope our response helps address your concerns. We also look forward to the subsequent discussion which may further help to solve the current issues.

W1 & Q1. Is memory–action misalignment specific to MemoryVLA?

The memory-action missalignment is a general problem to memory-agument VLA models, not only specific to MemoryVLA

To validate our claim, we ran the same original/replaced-history test on the released MemER and MME-VLA checkpoints. We used four history-dependent RoboMME tasks. These checkpoints were trained on RoboMME, so MIKASA would be out of their training distribution. Within each scene pair, the two histories specify different correct targets, while the current observation and robot state at handover are identical. Both models keep their native memory updates throughout. The results are in App. E.7, Table 18.

Table R2.1. Paired-history evaluation of released checkpoints. Both = success under the original and the replaced history in the same scene.

ModelTaskScenesOriginal ↑Replaced ↑Both ↑
MME-VLAButtonUnmask2015%30%0%
MME-VLAVideoUnmaskSwap2025%25%0%
MME-VLAVideoUnmask5026%20%0%
MME-VLAMoveCube5048%38%4%
MemERButtonUnmask2070%75%55%
MemERVideoUnmaskSwap2025%10%5%
MemERVideoUnmask2060%50%35%
MemERMoveCube2055%75%40%

MME-VLA shows strong misalignment.

  • On the three container tasks, it picked the same container under both histories in all 90 scene pairs, even though the correct target differed. As a result, it never succeeded under both histories.
  • On MoveCube, it succeeded under the original history in 24 scenes. In 16 of these 24, it repeated the original operation after the history demonstrated a different one.

MemER shows partial misalignment.

  • It follows the history in many scenes, but it still kept the old target in 6 scene pairs across the three container tasks.
  • We saw no such persistence among its originally successful MoveCube scenes.

So the memory-action missalignment problem is not specific to MemoryVLA, but its severity depends on the architecture. MemER converts history into explicit textual subgoals, which may partly explain why it suffers less. This also matches the smaller gain MemAlign brings to MemER (+1.88 pp, Table 3): there is less misalignment left to correct.

W2 & Q2. Component usage and transferability of the complete MemAlign framework

We thank the reviewer for this question. We agree that the original manuscript did not clearly state which components were used in each setting. Below we (1) clarify the role of each component, (2) report a new transfer of the complete framework to MemER, and (3) clarify the task-dependent use of counterfactual supervision on RoboMME.

(1) Role of each component.

  • Counterfactual supervision and history-aware low-rank modulation form the core of MemAlign. Their individual contributions are isolated in the matched ablation (Table 4).
  • Visual rebinding is an optional component. It connects the target identified from history to its location in the current observation. We use it for color-dependent tasks such as RememberColor. In those tasks, the DINO- and SigLIP-based features of the evaluated policies retain useful spatial information but sometimes discriminate colors poorly. Correctly remembering the cue therefore does not guarantee correct grounding of the corresponding object.

(2) Transfer of the complete framework (Q2). In the original submission, MemER used only counterfactual supervision, and MME-VLA used counterfactual supervision together with history-aware modulation. For MemER, transferring the supervision objective alone was a simple adaptation that preserved its native subgoal hierarchy. We acknowledge, however, that this showed only the transferability of the supervision strategy, not of the complete framework. In the revision, we adapt all three components to MemER (Sec. 5.3, App. D):

  • counterfactual supervision;
  • history-aware modulation of the low-level action policy;
  • visual rebinding, implemented as 2-D grounding with learned joint-action fusion to fit MemER's joint-action interface. We explicitly distinguish this from the metric rebinding used for MemoryVLA.

All baselines are rerun locally under the same protocol as their adapted counterparts.

Table R2.2. RoboMME average success (%) (Table 3 in the revision).

ModelComponentsBaseline+ MemAlignΔ (pp)
MemoryVLAmodulation + CF on 7/16 tasks8.2511.50+3.25
MemERCF + modulation + rebinding41.8743.75+1.88
MME-VLACF + modulation45.3846.50+1.12

(3) Counterfactual supervision on RoboMME (Tables 6–7). We thank the reviewer for highlighting the unclear annotation of counterfactual supervision. Our existing MemoryVLA-based RoboMME experiments use counterfactual supervision on seven tasks: all four Permanence tasks (VideoUnmask, ButtonUnmask, VideoUnmaskSwap, and ButtonUnmaskSwap), and three Reference tasks (PickHighlight, VideoPlaceButton, and VideoPlaceOrder). For these tasks, alternative histories share matched current observations, instructions, and robot states, and each history has a corresponding expert action. VideoRepick and all Counting and Imitation tasks use action fitting and history-aware modulation without the auxiliary counterfactual objective. All 16 tasks share the same adapted policy, and the reported results already reflect this task-dependent use of counterfactual supervision. We will make the task coverage explicit in Tables 6–7 and the accompanying text. Extending counterfactual construction to the remaining tasks is left for future work.

Q3. Inference latency, memory overhead, and training cost

We thank the reviewer for this question. Per you suggestion, we further provided the measurements below, and included this experiment results in App. F.2 (Table 19).

Inference latency and memory. MemAlign adds only a modest inference overhead relative to its gains. Compared with MemoryVLA, it adds about 50 ms per query (+11%) and about 0.8 GiB of peak GPU memory (+5%). In the same settings, it raises exclusive target completion from 2% to 98% on RC5 and from 14% to 90% on ShellGameTouch (Fig. 3). Both models are measured on the same NVIDIA L40 GPU (batch size 1, BF16, 10 DDIM steps, full memory banks), and each query generates one 16-step action chunk.

Table R2.3. Inference latency per 16-step action chunk (mean ± std over 300 queries) and peak allocated GPU memory. Extra memory is computed before rounding.

TaskModelLatency (ms)Peak (GiB)Extra (GiB)
RC5MemoryVLA455.4 ± 8.016.067—
RC5MemAlign509.0 ± 9.816.886+0.819
ShellMemoryVLA447.9 ± 8.716.071—
ShellMemAlign496.2 ± 4.316.882+0.810

Training cost. MemAlign is a lightweight post-training method. The pretrained backbone stays frozen, and only the added modules are trained. These modules have 3–5M parameters in the RoboMME, MME-VLA, and MemER adaptations: 4.6M, 3.1M, and 3.6M, respectively. This is less than 1% of the 7B-scale backbone. Training takes 224–15,000 updates, and the final adaptations run on a single NVIDIA L40S GPU (App. B and G.1). Applying MemAlign to an existing memory-augmented VLA therefore needs neither backbone retraining nor large-scale compute.

Dear Reviewer 1Cz8,

Thank you again for your time and constructive feedback. We would like to briefly follow up on our response and revised manuscript.

To address your concerns, we conducted misalignment experiments on MME-VLA and MemER, transferred the complete MemAlign framework to MemER, and reran the matched component ablation. We also clarified the core and optional components, specified the task-dependent use of counterfactual supervision on RoboMME, and provided latency, memory, and training-cost details.

We would appreciate your feedback on whether these additions address your concerns, or whether there are remaining questions we could clarify. If the additional evidence resolves your main concerns, we would be grateful if you could consider updating your score.

Best regards,

The Authors

Reviewer_ZzhaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The proposed MemAlign method is a post-training approach for memory-augmented VLA policies. It aims at solving the problem "the current image and instruction are compatible with more than one target, the history identifies the right one, and the policy still acts on the wrong object". On a controlled MIKASA-Robo protocol, exclusive completion of the history-specified target rises from 13% for the released MemoryVLA checkpoint to 87%. On RoboMME the same backbone, the proposed approach still works well.

Critical strengths
  1. The failure case is concrete, and Figure 1 distinguishes “the action changed” from “the action changed to the target this history specifies.” Original, replaced, and erased histories are the right intervention for that claim. Based on these cases, the proposed MemAlign method acheives remarkable performance, compared with MemoryVLA (Figure 3). This demonstrates the effectiveness of the proposed method.

  2. This method performs very well overall, enhancing the model's memory across several simulation problems.

Critical weaknesses
  1. In my view, VLA papers that lack experiments on actual hardware—failing to demonstrate performance in real-world scenarios or provide data on metrics like operating frequency—carry very little persuasive weight. For this reason, I can accept the rejection of this paper.

  2. The generalization claim is not fully supported. The abstract says MemAlign consistently improves different memory-augmented VLAs. On RoboMME, MemoryVLA improved from 8.25% to 10.88%—an increase of 21 successful cases out of 800. Meanwhile, performance on the "Counting" task declined from 9.5% to 6.5%.

Questions for authors
  1. The evaluation appears to lack multiple experimental runs; I noticed that some success rates in your MIKASA-Robo tasks reached 100%. Could you please report the average success rate and variance based on multiple independent runs?

  2. How to construct counterfactual supervision effectively? It seems require simulator target coordinates. Does this make it difficult to transfer the method to real-world scenarios?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Zzha

We thank the reviewer for the insightful comments! In the following, we provide our point-by-point response, which we hope helps address your concerns. We also look forward to the subsequent discussion which may further help to solve the current issues.

Q1. Multiple evaluation runs and variance

Per your suggestion, we re-evaluated each trained policy on three independent evaluation blocks. Each block uses a different evaluation seed and a different set of scenes, with 50 episodes per task. Table 1 of the revision now reports the mean ± sample standard deviation over the three blocks.

Table R1.1. Exclusive target completion (%) on MIKASA-Robo, mean ± std over three evaluation blocks.

MethodRC3RC5RC9ShellGameTouchAverage
MemoryVLA30.67 ± 3.062.00 ± 2.0010.67 ± 4.6216.67 ± 4.6215.00 ± 2.00
MemAlign (Ours)98.67 ± 1.1597.33 ± 2.3166.00 ± 5.2989.33 ± 2.3187.83 ± 2.52
  • The 100% entries. The 100% on RC3 came from a single block. Averaged over three blocks, the result is 98.67 ± 1.15.
  • Gain vs. variance. MemAlign improves over MemoryVLA by 72.83 pp on average. The block-level standard deviations are reported separately in Table R1.1.
  • Scope. These blocks measure evaluation variance for one trained policy. They do not measure variance across training initializations. Appendix G.2 states this explicitly. The component ablation (Table 4) is reported in the same way.

W1 & Q2. Real-robot experiments and counterfactual supervision

Real-robot setup (Sec. 5.4, App. F). We added real-robot experiments on two tasks, ShellGameTouch and RememberColor3. The robot is a PiPER-X arm with a fixed RealSense D435 camera and a wrist-mounted D405 camera. In both tasks, the cue disappears before the robot acts, and the robot must then touch one of three candidates. The baseline is MemoryVLA adapted to the same embodiment with the same 108 demonstrations per task (Stage 1 only, i.e., without history-aware modulation or counterfactual supervision). All methods use the same held-out layouts.

Table R1.2. Real-robot exclusive target completion (successes / 10 trials).

TaskMethodOriginal ↑Replaced ↑Erased†
ShellGameTouchMemoryVLA4/102/103/10
ShellGameTouchMemAlign6/104/102/10
RememberColor3MemoryVLA1/102/101/10
RememberColor3MemAlign5/103/101/10

†Erased removes the cue and is scored against the original target. It is a diagnostic, not a success metric.

  • ShellGameTouch. MemAlign raises Original completion from 4/10 to 6/10 and Replaced completion from 2/10 to 4/10. Its original-target completion drops to 2/10 when the cue is erased, consistent with sensitivity to the cue history.
  • RememberColor3. MemAlign raises Original completion from 1/10 to 5/10 and Replaced completion from 2/10 to 3/10. Its original-target completion under erased histories is 1/10. Failures generally fall into two categories: the gripper is not correctly oriented toward the target object, or it is oriented toward the target but fails to make proper contact. This can be observed in the videos provided in the supplementary materials.
  • Scope. This is a small feasibility study, not a real-world benchmark. Replaced-history redirection on hardware (4/10, 3/10) remains limited. We state this in the limitations.

Operating frequency. The policy predicts 16-step joint-target chunks on a 10 Hz temporal grid, covering a 1.6 s horizon. In the 16-step latency benchmark on L40, MemAlign takes 509.0 ± 9.8 ms per query versus 455.4 ± 8.0 ms for MemoryVLA on RC5 (+11.8%), and 496.2 ± 4.3 ms versus 447.9 ± 8.7 ms on ShellGameTouch (+10.8%) (App. F.2, Table 19). The measured real-robot RC3 query latency is 497.65 ms. At deployment, RTC overlaps inference with execution of buffered actions, while local interpolation preserves the predicted trajectory timing and supplies joint-target commands at 30 Hz. This separates the 30 Hz command publication rate from the approximately 2 Hz compute-limited policy query rate; it does not imply 30 Hz visual replanning.

Counterfactual supervision without simulator coordinates. On the real robot, we build counterfactual groups from teleoperated demonstrations. For a given object layout and starting pose, we record one demonstration per cue, each continuing to its own target. At the shared decision point, we keep the current observation and proprioception and substitute the recorded cue history. The demonstration recorded with that cue supplies the action continuation. As described in Sec. 5.4, supervision uses three valid color-history/action branches at a verified common decision state. Training combines action fitting with centered action-contrast alignment under shared sampling noise. Simulator target coordinates are used neither for real-robot supervision nor at deployment.

W2. Generalization claim and the Counting regression

We agree that the original wording overstated the generality of our results, and we have revised the claims accordingly.

Revised claim. The abstract and introduction no longer say "consistently improves". They now say that MemAlign improves average success on RoboMME for all three memory-augmented VLAs we tested (Table 3). Each comparison uses the same locally rerun evaluation protocol for the baseline and for MemAlign. The gains are moderate: +3.25 pp for MemoryVLA, +1.88 pp for MemER, and +1.12 pp for MME-VLA. They are largest on Reference tasks, e.g., +8.5 pp for MemoryVLA (Table 15). These are the tasks where the history determines which target to act on, which is the failure MemAlign is designed to fix. We also state in the paper that the gains are not uniform across categories.

Why Counting dropped. We examined the five PickXtimes episodes where MemoryVLA succeeded and MemAlign failed.

  • Three were termination errors: the policy stopped too early or kept going after the required number of repetitions.
  • Two were grasp failures, after which the policy stopped before completing the task.

In all five, the failure was in execution or in deciding when to stop. None showed the policy forgetting how many repetitions it had already done.

Partial fix. We added targeted expert demonstrations for these failure modes, collected from training seeds only, and fine-tuned for 2,000 more updates. Counting success rose from 13/200 (6.5%) to 18/200 (9.0%). This is still slightly below MemoryVLA's 19/200 (9.5%). The overall RoboMME result changed from 87/800 (10.88%) to 92/800 (11.5%). Appendix E.2, Table 15 reports all 16 tasks with 50 episodes per task, totaling 92/800 for MemAlign and 66/800 for MemoryVLA. Within the revision period we could only collect a small amount of extra data. We list the remaining Counting gap as a limitation (App. G.2). The data and training details are in App. B.5, and the results are in App. E.2, Table 15.

Table R1.3. Counting results on RoboMME (successful episodes / 50).

TaskMemoryVLAMemAlign (before)MemAlign (after)Δ vs. MemoryVLA (pp)
BinFill644−4
PickXtimes11610−2
SwingXtimes2120
StopCube022+4
Total / 20019 (9.5%)13 (6.5%)18 (9.0%)−0.5

Dear Reviewer Zzha,

Thank you again for your time and constructive feedback. We would like to briefly follow up on our response and revised manuscript.

To address your concerns, we added real-robot evaluations on RememberColor3 and ShellGameTouch, reported variability across three evaluation blocks, and analyzed the Counting regression with additional training results. We also clarified how counterfactual supervision is constructed from real demonstrations and qualified the scope of our generalization claims.

We would appreciate your feedback on whether these additions address your concerns, or whether there are remaining questions we could clarify. If the additional evidence resolves your main concerns, we would be grateful if you could consider updating your score.

Best regards,

The Authors

Dear Reviewer Zzha,

As the discussion period ends today, we wanted to follow up one last time. For convenience, here is where each of your original concerns is addressed in the revised manuscript:

  • Real-robot evidence: new evaluations on ShellGameTouch and RememberColor3 on a PiPER-X arm (Sec. 5.4, App. F); latency analysis in App. F.2, Table 19
  • Variability: mean ± std over three independent evaluation blocks (Table 1; scope stated in App. G.2)
  • Counting regression: failure analysis and additional training, which narrows the Counting gap to 18/200 vs. 19/200 for MemoryVLA (App. B.5; App. E.2, Table 15; remaining gap listed as a limitation in App. G.2)
  • Counterfactual supervision from real demonstrations: constructed from teleoperated demonstrations without simulator coordinates (Sec. 5.4)
  • Generalization claims: "consistently improves" removed from the abstract and introduction; claims now scoped to the three tested VLAs (Table 3)

Even a brief note on whether these address your concerns would be very helpful to us. If the revision resolves them, we would be grateful if you would consider updating your score.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback. We are encouraged that the reviewers found the problem of memory–action misalignment clear and well motivated (Zzha, 1Cz8, RKCn), the original/replaced/erased-history protocol a well-designed diagnostic (Zzha, 1Cz8, RKCn), and the improvement on MIKASA-Robo substantial (Zzha, RKCn).

The reviewers raised several shared concerns. We summarize our main revisions below. Changes in the manuscript are marked in blue.

1. Real-robot evaluation (Zzha, RKCn). We added real-robot experiments on ShellGameTouch and RememberColor3 using a PiPER-X arm (Sec. 5.4, App. F). We also explain how counterfactual supervision is constructed from real demonstrations without simulator coordinates.

2. Misalignment beyond MemoryVLA (1Cz8). We ran the paired-history test on the released MME-VLA and MemER checkpoints. MME-VLA selected the same container under both histories in all 90 scene pairs across three tasks, even though the correct targets differed. MemER showed milder but recurring misalignment. The problem is therefore not specific to MemoryVLA.

3. Clear definition of MemAlign and full transfer (1Cz8, RKCn). We now define counterfactual supervision and history-aware modulation as the core of MemAlign, and visual rebinding as an optional component. We also state which components each experiment uses (revised Table 6). We transferred all three components to MemER. Under locally rerun baselines, MemAlign improves average RoboMME success for all three backbones: MemoryVLA from 8.25% to 11.50%, MemER from 41.87% to 43.75%, and MME-VLA from 45.38% to 46.50% (Table 3).

4. Matched ablation (1Cz8, RKCn). We reran the ShellGameTouch component ablation under the main experimental configuration (Table 4). Removing history-aware modulation reduces success under both histories from 84.67% to 2.67%. Removing counterfactual supervision reduces it to 52.00%.

5. Fair comparison across protocols (RKCn). We evaluated MemoryVLA and MemAlign under both our controlled protocol and the original protocol. MemAlign improves average success under both: from 15.00% to 87.83% (controlled) and from 39.67% to 50.50% (original). This analysis also explains the gap between published and controlled MemoryVLA results.

6. Statistical reliability and cost (Zzha, 1Cz8). MIKASA results are now reported as mean ± std over three evaluation blocks (Table 1). MemAlign adds about 11% inference latency and 0.8 GiB of GPU memory, and it trains only 3–5M added parameters on a frozen backbone (App. F.2).

7. Revised claims (Zzha, RKCn). We removed "consistently improves" from the abstract and introduction. We now state that MemAlign primarily improves history-dependent target selection. The limitations section explicitly discusses the uneven gains across RoboMME categories and the reliance on controlled cue timing.

We address each reviewer's comments in detail in the individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.