- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes IMPRINT, a policy-agnostic latent memory framework that compresses visual, action, and proprioceptive histories into fixed-size memory tokens. A two-stage procedure first trains the memory encoder through historical reconstruction and normalized task-progress prediction, then jointly adapts the encoder and a pretrained VLA through action supervision. The method is evaluated on RMBench, a customized RoboCasa benchmark, and real-world manipulation tasks, with additional analyses of memory capacity, recurrent encoding, training objectives, learned representations, and adaptation across VLA backbones.
Critical strengths
- The paper presents a clear and well-motivated formulation that distinguishes memory encoding from memory learning. This framing consistently guides the methodology and experimental design, making the paper easy to follow.
- The paper provides extensive ablations and representation analyses that go beyond aggregate task success. These studies offer useful insights into how memory capacity and supervision affect historical information retention and its use for control.
Critical weaknesses
While the paper provides systematic and extensive analyses that offer valuable insights into memory encoding and learning, the methodology largely combines established latent memory bottlenecks, history-reconstruction objectives, and task progress estimation, leaving limited technical novelty beyond their integration.
Questions for authors
- Low-level action/state inputs are low-dimensional relative to the proposed latent memory, so reconstructing these directly available signals may require little temporal abstraction. What if raw action/state histories were provided directly to the policy without incorporating them into latent memory? Such an analysis would help clarify how integrating these streams into memory affects visual history retention or task progress estimation.
- Action/state histories may provide similar information (especially in an absolute joint action space), yet the reported analyses suggest different effects for these two streams (e.g., attention patterns in Figure 9 and ablations in Table 20). Could the authors explain this difference?
- Could the authors add a matched baseline that bypasses the memory bottleneck and directly attends to the full multi-stream history, reporting success and latency across history lengths? What specific shortcuts arise, and does IMPRINT mitigate them?
- Can the memory encoder be pretrained on a broader dataset in Stage 1 and transferred to target tasks through Stage 2? Such additional experiments could further strengthen the practical value of the proposed framework.
5 comments
Response to Reviewer xHCF (Part 2)
Q2. Why action and state histories have different effects
Although correlated, these streams capture different information: actions specify commanded targets of control, while proprioceptive states record the actual joint state the robot has achieved. Even in an absolute joint action space, a commanded target can differ from the subsequently observed configuration because of tracking delays, contact constraints, or incomplete execution.
Action history therefore preserves the intended command sequence, which can help the policy recall and reuse previous commands. State history captures how the robot actually moved and which configurations it reached. Jointly encoding both makes discrepancies between intended and executed motion available to the policy, potentially helping it account for execution errors when predicting subsequent actions. This distinction is particularly relevant to repeated manipulation, where errors can accumulate across operations.
The ablations support this complementary role: adding state history to vision-plus-action memory improves Block Ranking success from 72% to 83%. Figure 9 further suggests that stream reliance depends on training supervision: E2E emphasizes state history even when the decision requires an initial visual color cue, whereas the pretrained variants attend to the relevant initial frames. Thus, our multi-stream inputs provide useful execution context and an appropriate training recipe is also needed to avoid one stream (e.g. state) becoming a source of spurious correlations under insufficient supervision (revised manuscript Table 21).
Q3. Comparing with a policy trained on full history without the bottleneck
Naively bypassing the memory bottleneck makes both training and full-history inference prohibitively expensive within our available compute budget. After downsampling 1,451 timesteps to 726, three camera views with 256 SigLIP tokens per image yield 557,568 historical visual tokens, or 2,178× IMPRINT’s 256 memory tokens, producing approximately 558,450 policy tokens in total.
Even at batch size 1, explicitly materialized dense attention would require 290.4 GiB for the boolean mask and 9.1 TiB per layer for eight-head FP32 attention logits, exceeding single-GPU memory during inference; training introduces further memory and computation costs. Memory-efficient attention can avoid these explicit allocations, but retains quadratic attention computation. These costs preclude testing this naive baseline within our budget and motivate compressing history before policy processing.
The growing context also increases inference latency. Physical Intelligence’s MEM documents the related problem for naive visual-history concatenation: latency rapidly exceeds its 300 ms reference budget on an H100 with four camera streams (Section III-C, Figure 3). These training and deployment constraints motivate compressing history before policy processing.
We therefore cannot evaluate this naive full-stream baseline within our compute budget. The counterfactual history intervention, linear probes, and attention analysis provide complementary evidence about shortcut learning in E2E, frozen, and jointly adapted memory (our response to Reviewer Xia1, W1; manuscript Table 1 and Appendix E.4).
Q4. Broader Stage 1 pretraining and transfer
We evaluate this by pretraining IMPRINT’s memory encoder on RoboTwin + RMBench in Stage 1, followed by action finetuning exclusively on RMBench target-task demonstrations in Stage 2. We compare this with RMBench-only Stage 1 pretraining, keeping the Stage 2 data unchanged. This ablation is performed with the recurrent memory encoder.
Table II. Broader Stage 1 pretraining. Target-task success rates (%) after Stage 2, using recurrent IMPRINT (manuscript Table 30).
| Training data / Evaluation task | Target-only pretraining | Broader mixed-data pretraining |
|---|---|---|
| Stage 1 data | RMBench | RoboTwin + RMBench |
| Stage 2 data | RMBench | RMBench |
| Observe and Pickup | 6 | 12 |
| Rearrange Blocks | 99 | 100 |
| Put Back Block | 100 | 100 |
| Swap Blocks | 100 | 100 |
| Swap T | 100 | 100 |
| Battery | 35 | 45 |
| Block Ranking | 84 | 88 |
| Cover Blocks | 100 | 100 |
| Press Button | 31 | 35 |
| Average | 72.8 | 75.6 |
Broader Stage 1 pretraining improves target-task success from 72.8% to 75.6%, showing that IMPRINT benefits from additional pretraining experience and retains these gains after target-task finetuning. This supports the practical value of separating memory pretraining from action finetuning. We leave scaling Stage 1 pretraining to larger and more diverse simulation and real-world datasets, as well as evaluating adaptation of the pretrained memory to unseen tasks and embodiments, to future work.
Response to Reviewer xHCF (Part 3)
Thank you again for your thoughtful feedback and we hope our response addresses your concerns. We are happy to discuss more if you have any remaining questions!
Best regards,
Authors
Response to Reviewer xHCF (Part 1)
We thank reviewer xHCF for the thoughtful review and for highlighting the distinction between memory encoding and memory learning. We address the associated questions of limited novelty, the memory encoding and difference between action and state history below.
W1. Limited technical novelty beyond integration of latent memory bottlenecks, history-reconstruction objectives, and task progress estimation
We respectfully disagree that IMPRINT’s novelty is limited to combining existing components. Established latent bottlenecks and prior memory supervision objectives do not, by themselves, provide the memory-learning formulation or capabilities demonstrated in our paper.
IMPRINT’s technical contribution is a policy-independent formulation for learning what a multi-stream memory bottleneck must preserve, with three distinctions:
Direct supervision of shared multi-stream memory. We jointly reconstruct historical visual, action, and proprioceptive information and predict task progress from one bottleneck. Unlike Workspace Models’ VLM-selected visual targets, our design makes supervision come directly from recorded interactions, without a separate history-selection mechanism.
Two-level compression trained for history retention. We reduce visual redundancy before recurrently integrating all streams. Historical reconstruction supervises the resulting memory, learning to extract critical information from both visual compression and recurrent updates.
Separation of memory formation from policy-specific readout. Memory is pretrained without an action policy, then accessed through cross-attention and FiLM adapters inside the VLA’s action expert. Joint adaptation connects the shared representation to control while preserving a common memory interface across different backbones, which has not been shown in prior work.
Q1. Feeding action/state history directly into the policy
Although action/state vectors are low-dimensional at each timestep, using their history requires integrating temporal relationships with visual events. Following Reviewer xHCF’s suggestion, we evaluate a baseline combining visual memory with direct action/state history.
In Stage 1, only visual features and language enter the 256-token recurrent memory bottleneck, with action/proprioception inputs and their reconstruction losses removed. In Stage 2, causal action and proprioceptive histories are independently projected to the π₀.₅ action-expert dimension, interleaved as (a0,q0,a1,q1,…), and appended directly to the action expert’s memory-conditioning sequence, bypassing the memory encoder.
Table I. Ablations of Direct Action/State History vs IMPRINT on RMBench with Recurrent Memory Encoder (manuscript Table 22).
This baseline achieves 64.4% versus 72.8% for recurrent IMPRINT, while its uncompressed action/state token count grows with episode length. These results support the benefit of jointly encoding visual, action, and state histories into a compact memory representation.