acceptodds
Under review as a conference paper at ICLR 2027

Memory Without Curation: Learning Multi-Stream Latent Memory for Robot Policies

Abstract

It is widely appreciated that robotic policies require long-term memory to perform complex tasks. However, directly conditioning policies on all past observations leaves policy learning susceptible to learning spurious shortcuts, and introduces large latency at policy deployment. As such, today’s memory-augmented policies curate what to remember, either by retaining only recent frames, selected keyframes, or textual summaries, or by learning latent representations of the visual stream alone. In this work, we propose an alternative approach IMPRINT, a policy-agnostic latent memory framework that augments vision-language-action (VLA) models with multi-stream memory without explicitly constraining what the policy should remember. IMPRINT compresses different streams into a latent memory that is updated incrementally at inference, enabling low latency. To avoid shortcut learning, IMPRINT adopts a two-stage training procedure, first supervising memory to reconstruct multiple streams (observations, actions, proprioception) of past information, as well as estimate current task progress, followed by end-to-end finetuning, jointly with the policy, on action prediction. Across both simulation and real-world tasks demanding memory, IMPRINT improves average success over the base VLA by at least 26%, outperforming popular alternatives, and maintaining lowest inference latency among memory-augmented VLAs. Through careful ablations, we demonstrate the utility of incorporating diverse information streams into robotic memory, validating the advantages of foregoing history curation. Project website: https://iclr2027imprint.github.io/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-language-action models, robot memory, long-horizon manipulation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xHCFOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes IMPRINT, a policy-agnostic latent memory framework that compresses visual, action, and proprioceptive histories into fixed-size memory tokens. A two-stage procedure first trains the memory encoder through historical reconstruction and normalized task-progress prediction, then jointly adapts the encoder and a pretrained VLA through action supervision. The method is evaluated on RMBench, a customized RoboCasa benchmark, and real-world manipulation tasks, with additional analyses of memory capacity, recurrent encoding, training objectives, learned representations, and adaptation across VLA backbones.

Critical strengths
  • The paper presents a clear and well-motivated formulation that distinguishes memory encoding from memory learning. This framing consistently guides the methodology and experimental design, making the paper easy to follow.
  • The paper provides extensive ablations and representation analyses that go beyond aggregate task success. These studies offer useful insights into how memory capacity and supervision affect historical information retention and its use for control.
Critical weaknesses

While the paper provides systematic and extensive analyses that offer valuable insights into memory encoding and learning, the methodology largely combines established latent memory bottlenecks, history-reconstruction objectives, and task progress estimation, leaving limited technical novelty beyond their integration.

Questions for authors
  • Low-level action/state inputs are low-dimensional relative to the proposed latent memory, so reconstructing these directly available signals may require little temporal abstraction. What if raw action/state histories were provided directly to the policy without incorporating them into latent memory? Such an analysis would help clarify how integrating these streams into memory affects visual history retention or task progress estimation.
  • Action/state histories may provide similar information (especially in an absolute joint action space), yet the reported analyses suggest different effects for these two streams (e.g., attention patterns in Figure 9 and ablations in Table 20). Could the authors explain this difference?
  • Could the authors add a matched baseline that bypasses the memory bottleneck and directly attends to the full multi-stream history, reporting success and latency across history lengths? What specific shortcuts arise, and does IMPRINT mitigate them?
  • Can the memory encoder be pretrained on a broader dataset in Stage 1 and transferred to target tasks through Stage 2? Such additional experiments could further strengthen the practical value of the proposed framework.
5 comments
AuthorsOpenReview ↗

Response to Reviewer xHCF (Part 1)

We thank reviewer xHCF for the thoughtful review and for highlighting the distinction between memory encoding and memory learning. We address the associated questions of limited novelty, the memory encoding and difference between action and state history below.

W1. Limited technical novelty beyond integration of latent memory bottlenecks, history-reconstruction objectives, and task progress estimation

We respectfully disagree that IMPRINT’s novelty is limited to combining existing components. Established latent bottlenecks and prior memory supervision objectives do not, by themselves, provide the memory-learning formulation or capabilities demonstrated in our paper.

IMPRINT’s technical contribution is a policy-independent formulation for learning what a multi-stream memory bottleneck must preserve, with three distinctions:

  1. Direct supervision of shared multi-stream memory. We jointly reconstruct historical visual, action, and proprioceptive information and predict task progress from one bottleneck. Unlike Workspace Models’ VLM-selected visual targets, our design makes supervision come directly from recorded interactions, without a separate history-selection mechanism.

  2. Two-level compression trained for history retention. We reduce visual redundancy before recurrently integrating all streams. Historical reconstruction supervises the resulting memory, learning to extract critical information from both visual compression and recurrent updates.

  3. Separation of memory formation from policy-specific readout. Memory is pretrained without an action policy, then accessed through cross-attention and FiLM adapters inside the VLA’s action expert. Joint adaptation connects the shared representation to control while preserving a common memory interface across different backbones, which has not been shown in prior work.

Q1. Feeding action/state history directly into the policy

Although action/state vectors are low-dimensional at each timestep, using their history requires integrating temporal relationships with visual events. Following Reviewer xHCF’s suggestion, we evaluate a baseline combining visual memory with direct action/state history.

In Stage 1, only visual features and language enter the 256-token recurrent memory bottleneck, with action/proprioception inputs and their reconstruction losses removed. In Stage 2, causal action and proprioceptive histories are independently projected to the π₀.₅ action-expert dimension, interleaved as (a0,q0,a1,q1,…)(a_0,q_0,a_1,q_1,\ldots), and appended directly to the action expert’s memory-conditioning sequence, bypassing the memory encoder.

Table I. Ablations of Direct Action/State History vs IMPRINT on RMBench with Recurrent Memory Encoder (manuscript Table 22).

TaskVisual memory + direct action/state history (%)IMPRINT recurrent (%)
Observe & Pick126
Rearrange Blocks7499
Put Back Block100100
Swap Blocks100100
Swap T91100
Battery4035
Block Ranking5484
Cover Blocks87100
Press Button2231
Task-balanced average64.472.8

This baseline achieves 64.4% versus 72.8% for recurrent IMPRINT, while its uncompressed action/state token count grows with episode length. These results support the benefit of jointly encoding visual, action, and state histories into a compact memory representation.

AuthorsOpenReview ↗

Response to Reviewer xHCF (Part 2)

Q2. Why action and state histories have different effects

Although correlated, these streams capture different information: actions specify commanded targets of control, while proprioceptive states record the actual joint state the robot has achieved. Even in an absolute joint action space, a commanded target can differ from the subsequently observed configuration because of tracking delays, contact constraints, or incomplete execution.

Action history therefore preserves the intended command sequence, which can help the policy recall and reuse previous commands. State history captures how the robot actually moved and which configurations it reached. Jointly encoding both makes discrepancies between intended and executed motion available to the policy, potentially helping it account for execution errors when predicting subsequent actions. This distinction is particularly relevant to repeated manipulation, where errors can accumulate across operations.

The ablations support this complementary role: adding state history to vision-plus-action memory improves Block Ranking success from 72% to 83%. Figure 9 further suggests that stream reliance depends on training supervision: E2E emphasizes state history even when the decision requires an initial visual color cue, whereas the pretrained variants attend to the relevant initial frames. Thus, our multi-stream inputs provide useful execution context and an appropriate training recipe is also needed to avoid one stream (e.g. state) becoming a source of spurious correlations under insufficient supervision (revised manuscript Table 21).

Q3. Comparing with a policy trained on full history without the bottleneck

Naively bypassing the memory bottleneck makes both training and full-history inference prohibitively expensive within our available compute budget. After downsampling 1,451 timesteps to 726, three camera views with 256 SigLIP tokens per image yield 557,568 historical visual tokens, or 2,178× IMPRINT’s 256 memory tokens, producing approximately 558,450 policy tokens in total.

Even at batch size 1, explicitly materialized dense attention would require 290.4 GiB for the boolean mask and 9.1 TiB per layer for eight-head FP32 attention logits, exceeding single-GPU memory during inference; training introduces further memory and computation costs. Memory-efficient attention can avoid these explicit allocations, but retains quadratic attention computation. These costs preclude testing this naive baseline within our budget and motivate compressing history before policy processing.

The growing context also increases inference latency. Physical Intelligence’s MEM documents the related problem for naive visual-history concatenation: latency rapidly exceeds its 300 ms reference budget on an H100 with four camera streams (Section III-C, Figure 3). These training and deployment constraints motivate compressing history before policy processing.

We therefore cannot evaluate this naive full-stream baseline within our compute budget. The counterfactual history intervention, linear probes, and attention analysis provide complementary evidence about shortcut learning in E2E, frozen, and jointly adapted memory (our response to Reviewer Xia1, W1; manuscript Table 1 and Appendix E.4).

Q4. Broader Stage 1 pretraining and transfer

We evaluate this by pretraining IMPRINT’s memory encoder on RoboTwin + RMBench in Stage 1, followed by action finetuning exclusively on RMBench target-task demonstrations in Stage 2. We compare this with RMBench-only Stage 1 pretraining, keeping the Stage 2 data unchanged. This ablation is performed with the recurrent memory encoder.

Table II. Broader Stage 1 pretraining. Target-task success rates (%) after Stage 2, using recurrent IMPRINT (manuscript Table 30).

Training data / Evaluation taskTarget-only pretrainingBroader mixed-data pretraining
Stage 1 dataRMBenchRoboTwin + RMBench
Stage 2 dataRMBenchRMBench
Observe and Pickup612
Rearrange Blocks99100
Put Back Block100100
Swap Blocks100100
Swap T100100
Battery3545
Block Ranking8488
Cover Blocks100100
Press Button3135
Average72.875.6

Broader Stage 1 pretraining improves target-task success from 72.8% to 75.6%, showing that IMPRINT benefits from additional pretraining experience and retains these gains after target-task finetuning. This supports the practical value of separating memory pretraining from action finetuning. We leave scaling Stage 1 pretraining to larger and more diverse simulation and real-world datasets, as well as evaluating adaptation of the pretrained memory to unseen tasks and embodiments, to future work.

AuthorsOpenReview ↗

Response to Reviewer xHCF (Part 3)

Thank you again for your thoughtful feedback and we hope our response addresses your concerns. We are happy to discuss more if you have any remaining questions!

Best regards,

Authors

Reviewer_Xia1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes IMPRINT, a policy-agnostic latent memory framework that compresses visual, action, and proprioceptive histories for VLA policies. It uses two-stage training: memory pretraining through history reconstruction and task-progress estimation, followed by joint adaptation with the VLA. Experiments on RMBench, RoboCasa, and three real-world tasks demonstrate substantial improvements in manipulation success with relatively low reported inference latency.

Critical strengths
  1. The method achieves promising improvements across two simulation benchmarks and three real-world tasks. Evaluating both memory-dependent and non-memory tasks is particularly valuable, demonstrating that memory augmentation can improve history-dependent behavior without degrading standard manipulation performance.

  2. The paper carefully investigates memory capacity, input streams, training objectives, and adaptation strategies. Representation analyses provide additional evidence that pretraining preserves task-relevant information. The recurrent encoder is efficient, and the memory pipeline integrates with multiple VLA backbones.

Critical weaknesses
  1. The paper's claim that pretraining avoids spurious shortcuts is supported mainly by representation probes and attention visualizations rather than direct evidence. Moreover, the two-stage approach adds 160k memory-pretraining updates without a compute-matched end-to-end baseline, making it difficult to separate the benefit of pretraining from the effect of additional training.

  2. Several RMBench baseline results are taken from prior work using different backbones and training pipelines. In particular, EventVLA is reported only in a simplified configuration. Furthermore, the RoboCasa comparisons omit competing latent-memory approaches. These gaps make it difficult to establish the size of IMPRINT’s advantage over the strongest existing memory methods.

Questions for authors
  1. Could the authors compare against the full EventVLA configuration, and explain why competing latent-memory methods are absent from RoboCasa?

  2. Does the reported latency include all online visual encoding and memory updates? Additionally, how do the real-world baselines perform under the same OOD object variations?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Xia1 (Part 1)

We thank reviewer Xia1 for recognizing the broad ablations and analysis of the paper and the importance of testing both memory-dependent and non-memory tasks. We agree that the training-budget comparison, competing memory baselines, and timing details deserve stronger treatment.

W1. Shortcut Analysis & E2E training with matched update budgets

Shortcut Analysis. The potential shortcut in Cover Blocks is to predict the uncovering target from spurious correlations in robot-state or motion histories without retaining the initial visual color order. Our linear probe supports this interpretation: E2E achieves only 19.94% color-order decoding accuracy, close to six-way chance (16.7%), compared with 83.31% for frozen Stage 1 memory and 91.53% after joint adaptation (Table 18).

To directly test whether historical information guides actions, we conduct a counterfactual history intervention at the first uncovering decision. For each of five states, we hold the current observation, robot state, and instruction fixed and supply each of the six possible block-order histories, yielding 30 evaluation cases. These histories imply three possible target locations, giving a chance accuracy of 33.3%. We measure both correct target selection and the change in the generated action chunk.

Table I. Counterfactual history intervention on Cover Blocks (manuscript Table 1; detailed protocol in Appendix E.4).

ModelCorrectAccuracyAction MSE ↑Arm-only MSE ↑
E2E10/3033.3%1.87e−7 ± 1.24e−81.97e−7 ± 1.37e−8
Frozen15/3050.0%0.155 ± 0.1240.180 ± 0.145
IMPRINT, jointly finetuned25/3083.3%0.347 ± 0.1720.403 ± 0.201

Action MSE measures the difference between the counterfactual-history and matched-history 50-step action chunks, using the executed 14-D action space; arm-only MSE excludes gripper commands. Values are reported as mean ± standard deviation across five states. Higher MSE indicates greater sensitivity to the historical intervention, while target accuracy measures whether that sensitivity produces the correct decision.

E2E’s near-zero MSE confirms that its actions remain virtually unchanged when the task-critical block-order history changes. Together with its chance-level target accuracy and the probe and attention analyses, this supports our interpretation that E2E overfits to spurious correlations and learns a shortcut instead of using the initial visual cue. Frozen Stage 1 memory already produces meaningful responses to history, while joint adaptation increases both action sensitivity and correct target selection, reaching 83.3% accuracy. The same pattern in arm-only MSE shows that these changes extend to arm motion. These results provide direct behavioral evidence that IMPRINT mitigates shortcut learning by preserving task-critical history and using it to guide actions.

Training-budget comparison. We compare E2E training and IMPRINT’s two-stage recipe using the same backbone, memory architecture, training data, and evaluation protocol. The original E2E baseline uses 100k updates, matching IMPRINT’s policy-training stage. We additionally extend E2E training to 260k updates, matching IMPRINT’s combined budget of 160k memory-pretraining updates + 100k policy-training updates. These controls match update count rather than exact FLOPs or wall-clock time because Stage 1 does not run the policy.

Table II. E2E training with matched total update budgets (manuscript Table 25). Each task uses 100 evaluation rollouts.

Budget / TaskOriginal E2EExtended E2EIMPRINT, full attention
Stage 1 updates——160k
Policy-training updates100k260k100k
Total updates100k260k260k
Observe & Pick12%6%9%
Rearrange Blocks90%85%100%
Put Back Block100%100%100%
Swap Blocks90%90%100%
Swap T45%41%100%
Battery27%30%41%
Block Ranking73%79%83%
Cover Blocks15%15%100%
Press Button5%8%21%
Task-balanced average50.8%50.4%72.7%

E2E performance appears to have saturated by 100k updates: extending training to 260k updates does not improve success (50.8% → 50.4%). At the same total budget of 260k updates, IMPRINT achieves 72.7% success, a 22.2-percentage-point improvement computed before rounding. This supports the benefit of decoupling memory pretraining from policy adaptation beyond simply training longer.

AuthorsOpenReview ↗

Response to Reviewer Xia1 (Part 2)

W2 & Q1. EventVLA evaluation & latent-memory methods on RoboCasa

EventVLA configurations and matched backbones. To address backbone differences, we reimplement two EventVLA variants on π0.5, using the same base policy as IMPRINT: EventVLA-fixed anchor-π0.5, which uses fixed visual anchors, and EventVLA-full-π0.5, which additionally incorporates learned Keyframe Evidence Memory, keyframe selection, and online memory updates. Their RMBench results are compared with recurrent IMPRINT below.

Table III. Comparisons between EventVLA and IMPRINT with the Same π0.5 Backbone.

TaskIMPRINT (recurrent)EventVLA-fixed anchor-π0.5EventVLA-full-π0.5
Observe & Pick6810
Rearrange Blocks998190
Put Back Block1006888
Swap Blocks1006780
Swap T1002390
Battery354045
Block Ranking849389
Cover Blocks100100100
Press Button3175
Average success (%)72.854.166.3

These comparisons control for the policy backbone and show that IMPRINT outperforms both the fixed-anchor and full EventVLA configurations.

RoboCasa latent-memory comparisons. Given our limited compute budget, we initially prioritized RMBench because it provides a shared benchmark across competing memory methods. Our RoboCasa evaluation complements this by assessing performance on both memory-dependent and non-memory tasks. Following the reviewer’s suggestion, we extend this comparison to HAMLET and MemoryVLA.

Table IV. Per-task success rates (%) on the customized RoboCasa benchmark.

TaskTypeHAMLETMemoryVLAIMPRINT recurrent
Retrieve fruitMemory-dependent687090
Retrieve oilMemory-dependent404680
Wash and returnMemory-dependent323096
Wash and return, same locationMemory-dependent525658
Sweeten hot chocolateMemory-dependent606064
Rinse cutting boardMemory-dependent004
Pan transferNon-memory545872
Organize condimentsNon-memory323874
Pick/place counter to sinkNon-memory908896
Place equal ice cubesNon-memory2220
Scrub cutting boardNon-memory363676
Frying-pan adjustmentNon-memory586070
Memory-dependent average—42.043.765.3
Non-memory average—45.347.068.0
Overall average—43.745.366.7

IMPRINT outperforms HAMLET and MemoryVLA on all 12 tasks, improving overall success by 23.0 and 21.3 percentage points, respectively. The gains span both memory-dependent and non-memory tasks, demonstrating an advantage beyond tasks explicitly designed to require memory. These RoboCasa results further support the benefits of IMPRINT over competing latent-memory methods (revised manuscript Tables 12 and 24).

AuthorsOpenReview ↗

Response to Reviewer Xia1 (Part 3)

Q2. Detailed Latency Breakdown & OOD Evaluation of Baselines

Latency measurement. Figure 6 reports trajectory-averaged policy-call latency: 130.8 ms for full attention and 122.8 ms for recurrent update. It includes all online visual encoding and memory updates.

We add a breakdown that includes all newly required visual encoding, memory updates for our full attention method and the recurrent method.

Table V. Detailed runtime breakdown (manuscript Table 28).

MethodOnline visual encodingMemory preparation/update/readoutTotal runtime per action call
IMPRINT, full attention19.5 ms21.9 ms130.8 ms
IMPRINT, recurrent15.0 ms8.4 ms122.8 ms

The recurrent design reduces memory overhead from 21.9 to 8.4 ms by maintaining a compact state instead of repeatedly encoding historical context. Most latency remains in the shared π0.5 action-generation backbone.

OOD baseline evaluation. We evaluate the real-world baselines on the same unseen object sets and towels used for IMPRINT, with the same task instructions, rollout counts, and success criteria, without additional OOD fine-tuning.

Table VI. ID / OOD success rates (%) for IMPRINT and baselines (manuscript Table 29). Each cell lists ID / OOD. Overall pools 72 trials per condition: 20 Dust Table, 20 Scan Bottle, and 32 Fold Cloth. It differs from the equal-task mean in the original real-world table because Fold Cloth has more trials.

MethodDust TableScan BottleFold ClothOverall
π₀30.0 / 25.025.0 / 15.021.9 / 12.525.0 / 16.7
HAMLET25.0 / 5.025.0 / 15.018.8 / 9.422.2 / 9.7
MemER40.0 / 5.035.0 / 10.037.5 / 15.637.5 / 11.1
Uniform Sampling50.0 / 30.050.0 / 20.046.9 / 21.948.6 / 23.6
Ours80.0 / 45.075.0 / 70.071.9 / 31.375.0 / 45.8

Under the same OOD variations, IMPRINT maintains the highest overall performance despite degradation across all methods, supporting the generalization of its learned memory beyond ID scenarios. MemER and HAMLET exhibit the largest drops, while uniform sampling remains the strongest baseline. IMPRINT is particularly robust on Scan Bottle, generalizing to bottles of different sizes. The remaining failures reveal challenges shared across methods: replacing round objects with thin, square objects in Dust Table reduces performance for most approaches, while changes in towel size and texture in Fold Cloth make precise folding without wrinkles difficult, especially for diagonal folds. Overall, IMPRINT’s sustained advantage under these shifts suggests that its learned memory remains useful beyond the ID setting.

Thank you again for your thoughtful feedback and we hope our response addresses your concerns. We are happy to discuss more if you have any remaining questions!

Best regards,

Authors

Reviewer_ozcSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes IMPRINT, a two-stage framework that augments vision-language-action (VLA) policies with a learned latent memory over multi-stream histories (visual, action, proprioceptive). Stage 1 pretrains a memory encoder via multi-stream reconstruction and progress estimation, without a policy in the loop. Stage 2 attaches the frozen-then-jointly-finetuned memory to a VLA action expert through a lightweight cross-attention + FiLM adapter. The central claim is that supervising memory directly (rather than through action loss alone or through curated keyframes) avoids spurious shortcuts and yields strong success with low inference cost. Experiments span RMBench (9 tasks), a customized RoboCasa benchmark (12 tasks), and three real-world bimanual tasks, with reported gains of +47.4% on RMBench and +50% real-world over the base VLA.

Critical strengths

The problem framing is genuinely well-motivated. The observation that memory-curation methods require task-specific priors about what to remember (keyframes, subtask boundaries, object-centric graphs) is a real limitation, and the paper correctly identifies that a learned latent bottleneck is a reasonable escape hatch. The "Encoding" vs. "Learning" decomposition in Section 1 is a clean way to structure the contribution.

The two-stage, policy-agnostic design has practical appeal. Decoupling memory pretraining from policy adaptation means one memory checkpoint can initialize π0 π0.5, and MolmoAct2 (Table 16), and the adapter-based injection into the action expert rather than the VLM context is a sensible engineering choice with supporting evidence in Figure 4 (heuristic memory in VLM context degrades non-memory tasks to 44.0%).

The efficiency numbers, if taken at face value, are competitive. 122.8 ms per 50-step action chunk with recurrent encoding versus 960 ms for EventVLA and 321.6 ms for MemoryVLA is a meaningful gap, and the per-capacity latency breakdown in Table 18 is a useful piece of systems analysis that many memory papers omit.

Critical weaknesses

The headline efficiency claim rests on a comparison that is not apples-to-apples, and the paper does not fully own up to it. Table 4 explicitly states that EventVLA and MemoryVLA results are "reported by prior work" on different backbones (QwenOFT / Qwen3-VL-4B) and different training pipelines. Yet Figure 6 places IMPRINT's 122.8 ms next to EventVLA's 960 ms and MemoryVLA's 321.6 ms on the same log-scale axis, and the abstract/contribution bullet advertises "lowest average latency of 122.8 ms per action chunk… among the memory baselines included in our latency comparison." The caveat is buried in Appendix A.2. A reader skimming Figure 6 will conclude IMPRINT is ~8× faster than EventVLA, when in fact the two numbers were not measured under the same protocol. This is not a fatal flaw, but it is the kind of framing that should be corrected before the claim is repeated in follow-up work.

The Stage 1 / Stage 2 decomposition is presented as cleanly additive, but the ablation data suggest otherwise. Figure 8 (left) reports 50.8% (E2E) → 58.8% (frozen memory) → 72.7% (joint adaptation). The +22-point jump from frozen to joint is attributed to "the action gradient in turn further refines memory for control." But Table 15 shows that on several tasks the frozen memory is at or below E2E: Press Button 21% (frozen) vs. 21% (E2E), Battery 41% vs. 27%, Block Ranking 68% vs. 73%. On these tasks Stage 1 contributes essentially nothing and the entire gain comes from Stage 2. The paper's narrative — "Stage 1 primes the bottleneck so that fine-tuning avoids spurious shortcuts" — would predict that frozen memory should already be a useful initialization across the board. It is not. The alternative explanation, that Stage 1 mainly serves as a warm-start for the encoder and the real work is done by joint action gradients, is not ruled out. A per-task scatter of "Stage 1 gain" vs. "Stage 2 gain" would materially clarify what the pretraining objective is actually buying.

The absolute performance on the tasks that most cleanly isolate memory is low, and the paper does not interrogate why. Press Button is the one task where the memory requirement is unambiguous (counting presses of visually identical buttons). IMPRINT reaches 21% (full attention) / 31% (recurrent) on RMBench (Table 10), while all baselines are ≤3%. A 21–31% success rate on a counting task, with 100 rollouts per task, is not a demonstration that the method "solves" memory — it is a demonstration that it is less broken than the alternatives. The failure analysis in Appendix E describes the failure modes qualitatively ("fails to press the right button enough times") but does not quantify how many failures are memory-tracking errors vs. manipulation execution errors. Without that decomposition, it is hard to tell whether the latent memory is actually retaining counts or whether the policy is simply more robust to distractor frames.

The recurrent encoder's claimed advantage is measured on a horizon where it barely matters, and the paper's own latency table undercuts the urgency of the design choice. Table 18 shows that at 1,451 frames the recurrent encoder saves 15.4–21.1 ms (10.7–14.2%) over full attention. But the full-attention variant reaches 72.7% on RMBench vs. 72.8% for recurrent — statistically indistinguishable. The recurrent encoder is justified primarily on RoboCasa (66.7% vs. 62.2%), where the gain is real but the mechanism is unexplained (the paper attributes it to "encoding efficiency," which is a latency argument, not an accuracy argument). If the accuracy difference on RoboCasa is the real motivation, the paper should analyze which tasks benefit and why; if the latency difference is the motivation, 15 ms on a 122 ms budget is a modest payoff for the added architectural complexity (gated delta blocks, FP32 matrix states, per-chunk cache management).

Questions for authors

On Figure 6, the EventVLA and MemoryVLA latencies are cited from prior work on different backbones. Can you re-measure at least EventVLA (or a re-implementation of its visual-anchor variant) on the same L40S / batch-size-1 / 50-step-chunk protocol used for IMPRINT? If the 960 ms figure shrinks substantially under a matched protocol, the efficiency claim in the abstract needs to be softened. If it does not, the matched measurement would substantially strengthen the paper.

On Table 15, for Press Button and Battery, frozen Stage 1 memory is at or below E2E. What does Stage 1 actually learn on these tasks that survives to Stage 2? A linear probe on the frozen memory for "number of presses so far" (analogous to the color-order probe in Table 17) would directly test whether the counting information is present but unusable by the frozen policy, or absent altogether.

The 4.5-point overall difference (66.7% vs. 62.2%) is larger than the RMBench difference and is not explained by latency. Is this a capacity effect (recurrent state is a different bottleneck than 256 query tokens), an optimization effect (gated delta training dynamics), or a task-specific artifact? A per-task breakdown exists in Tables 10–11, but the paper does not interpret it.

5 comments
AuthorsOpenReview ↗

Response to Reviewer ozcS (Part 1)

We thank the reviewer ozcS23 for the detailed feedback and for recognizing the motivation and practical value of separating memory learning from policy adaptation. We address the concerns about efficiency comparisons to baselines, roles of stage-1/stage-2 training, failure analysis of the results and the usefulness of the recurrent variants through the experiments and explanations below.

W1 & Q1. Latency Comparison with Matched Backbone

We agree that the original comparison combined different backbones and runtime settings, which could overstate the efficiency gap. To address this, we reimplement MemoryVLA and both EventVLA variants on π₀.₅ and measure all methods under the same protocol: one L40S GPU, batch size 1, three cameras, 50-action chunks, ten action-network evaluations, and synchronized policy-call timing.

Table I. Original reported runtimes and matched π₀.₅ measurements.

MethodOriginal published/tested latencyMatched π₀.₅ latency
π₀.₅ without memory—91.0 ms
IMPRINT, recurrent—122.8 ms
IMPRINT, full attention—130.8 ms
MemoryVLA321.6 ms230.6 ms
EventVLA, fixed anchors960 ms267.0 ms
EventVLA, full1,090 ms366.2 ms

The latency difference between EventVLA-π₀.₅ and EventVLA-QwenOFT primarily reflects their backbone and runtime choices: QwenOFT processes image and action-query tokens through its full Qwen3-VL-4B model using Hugging Face, whereas π₀.₅ caches the vision–language prefix and uses a smaller action expert with compiled JAX/XLA execution.

MemoryVLA’s latency difference between different backbones (π₀.₅ vs Prismatic-7B) is smaller: its main overhead comes from dual 256-entry memory banks, retrieval, CFG, online updates, and consolidation.

Our matched reimplementations retain the baseline memory operations. MemoryVLA includes both 256-entry banks, retrieval, classifier-free guidance, online updates, and token-merging consolidation; full EventVLA includes visual anchors and keyframe-memory operations.

IMPRINT retains its efficiency advantage under the matched protocol and backbone. Recurrent IMPRINT achieves 46.7%, 54.0%, and 66.5% lower latency than MemoryVLA, fixed-anchor EventVLA, and full EventVLA, respectively, while adding only 31.8 ms over π₀.₅ without memory. Full-attention IMPRINT also remains faster than all three baseline variants.

We report the matched online timing measurements in manuscript Table 23 and refine the abstract accordingly: **among the evaluated π₀.₅-based memory methods, recurrent IMPRINT has the lowest latency at 122.8 ms per 50-action chunk on an L40S GPU. The no-memory π₀.₅ baseline is 91.0 ms, giving 31.8 ms of added latency for recurrent IMPRINT. Figure 6 retains the original success–latency plot.

AuthorsOpenReview ↗

Response to Reviewer ozcS (Part 2)

W2. Contributions of Stage 1 and Stage 2 joint adaptation

We first clarify the frozen-memory success rates: 2% on Press Button and 20% on Battery, compared with 5% and 27% for E2E, respectively (revised manuscript Table 16). Following the reviewer’s request, we report Stage 1 gain as Frozen − E2E and Stage 2 gain as Joint − Frozen, in percentage points below and in revised manuscript Table 16.

Table II. Ablations over Different Training Stages

TaskE2E (%)Frozen memory (%)Joint adaptation (%)Stage 1 gain (pp)Stage 2 gain (pp)
Observe / Pick1279−5+2
Rearrange Blocks9099100+9+1
Put Back Block10010010000
Swap Blocks9099100+9+1
Swap T4599100+54+1
Battery272041−7+21
Block Ranking736883−5+15
Cover Blocks1535100+20+65
Press Button5221−3+19
Average50.858.872.7+8.0+13.9

The average gains are therefore 8.0 points from Stage 1 and 13.9 points from Stage 2, with a 21.9-point total improvement over E2E (Table 16).

We agree that frozen memory does not improve every task, but task success also depends on both memory and precise manipulation. On tasks where frozen memory underperforms, manipulation difficulty may limit the benefit of improved memory. Battery Task requires precise grasping otherwise the battery would roll away, while Block Ranking Task involves repeated pick-and-place operations where accumulated inaccurate placements can cause failures. Observe and Pick Up requires recognizing and grasping a wide variety of objects, including unseen instances in our evaluation. These demands in manipulation may limit the benefit of a frozen memory representation, so lower success alone does not establish that Stage 1 retains less useful history than E2E. Press Button remains difficult for both E2E and frozen memory (5% and 2%), while joint adaptation improves success to 21%.

Stage 1 nevertheless provides substantial gains on tasks requiring precise recall of earlier information. Swap T requires remembering the objects’ initial positions and orientations: frozen pretrained memory raises success from 45% to 99%, before joint adaptation. Cover Blocks requires retaining the initial color arrangement after the blocks are covered: success improves from 15% to 35% after Stage 1, then to 100% after Stage 2. The existing color-order probe further supports this interpretation that stage-1 is helpful for building a good memory representation to extract task-relevant information instead of overfitting to unrelated details: on the analyzed demonstrations, accuracy rises from 19.94% with E2E memory to 83.31% with frozen pretrained memory (Table 18). These results show that Stage 1 already preserves useful historical information, while joint adaptation helps the policy use and refine that representation for control.

We agree that Stage 1 also serves as a warm start for the memory encoder. We have clarified “priming the bottleneck” in the revised paper: Stage 1 establishes a representation that preserves useful historical information and provides a starting point for adaptation with the action objective. Our claim concerns how this representation supports subsequent joint optimization of memory and policy in Stage 2. The large overall performance gains should therefore be understood as the result of the complete two-stage process, rather than attributed to Stage 1 alone.

AuthorsOpenReview ↗

Response to Reviewer ozcS (Part 3)

W3 & Q2. What Stage 1 learns on Press Button Task and Its Failure Modes

We agree that task success alone does not establish whether counting information is present in memory. Following the color-order analysis in Table 18, we therefore train linear probes to predict cumulative press counts from memory representations. Frozen Stage 1 memory achieves 84.7% balanced accuracy, compared with 69.9% for E2E; joint Stage 2 adaptation further improves this to 91.6% (Table III). Thus, count information is already linearly decodable after Stage 1 and remains accessible after Stage 2, with improved decodability. The weaker frozen-policy success in Table 16 highlights that encoding count information and reliably using it for action prediction are distinct requirements.

Probe protocol. We use all 50 demonstrations, yielding 386 samples. Each sample is extracted immediately after a completed non-confirm press and labeled with the cumulative press count (1–9). We mean- and max-pool the 256 memory tokens, concatenate and standardize the pooled features, and train a single linear softmax classifier. Evaluation uses five-fold episode-grouped cross-validation, ensuring that samples from the same episode never appear in both classifier training and testing. These folds use demonstrations available during memory training, so this probe measures decodability rather than memory generalization to unseen trajectories. We report balanced accuracy, the mean recall across the nine counts, to account for fewer samples at higher counts.

Table III. Linear decoding of cumulative press count from memory. Entries report per-class recall.

Cumulative press countSamplesE2EFrozen Stage 1Joint Stage 2
15098.0%100.0%100.0%
25098.0%100.0%98.0%
35098.0%96.0%92.0%
44885.4%85.4%100.0%
54858.3%91.7%93.8%
64845.8%85.4%93.8%
74163.4%68.3%87.8%
83141.9%80.6%83.9%
92040.0%55.0%75.0%
Balanced accuracy38669.9%84.7%91.6%

Remaining failures. Press Button requires distinguishing successful presses from unsuccessful attempts, and switching or stopping at the correct count. IMPRINT achieves 21% success with full attention and 31% with recurrence, compared with at most 7% among the evaluated baselines (revised manuscript Table 11). To characterize the remaining failures, we categorize unsuccessful trajectories from 100 evaluation rollouts per encoder into counting/stopping errors and physical execution failures.

Table IV. Failure breakdown among unsuccessful Press Button rollouts.

EncoderCounting/stopping errorsExecution/manipulation failures
Full attention: 79 failures67/7912/79
Recurrent: 69 failures69/690/69

Counting/stopping errors account for 84.8% and 100% of failures, respectively. Together with the linear-probe results—84.7% count-decoding accuracy after Stage 1 and 91.6% after joint adaptation—this shows that IMPRINT learns to incorporate count-related information in the memory representations and substantially improves memory-dependent control. The failure breakdown identifies a concrete direction for further work: how to better translate this encoded information into more reliable action generation (e.g., generate better switching and stopping decisions during closed-loop execution).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.