- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper conditions a pretrained robot policy on recent scalar value predictions and adapts it through online reinforcement learning. The value history is encoded with proprioceptive history and supplied to a flow-matching policy initialized from $\pi$0.5. The method training proceeds via demonstration initialization, offline-RL, human-assisted collection, and autonomous learning that retains earlier interventions in replay. The paper’s idea is to use predicted values as policy memory. Experiments on three challenging physical tasks is presented to support the claim made in the paper.
Critical strengths
The central idea is appealing. The asynchronous implementation gives the idea practical usage. The action policy consumes previous value predictions while value inference proceeds separately, making the cache usable without requiring a current critic prediction before every action.
The strongest evidence is the consistency of the physical-task gains.
Improvements across deformable-object handling and contact-sensitive assembly is good.
The warmup ablation, earlier-critic substitutions and blind recovery annotation also provide more insight than final success alone.
Critical weaknesses
Removing the cache causes a large performance drop, so the main ablation does establish that the added input helps. What remains unclear is why it helps. Each cached prediction is computed from an observation history…. the oldest cached prediction may therefore expose the actor to information beyond the no-cache actor’s effective observation horizon. The paper does not specify those horizons….
Additional temporal context, learned features, and value-specific information remain plausible explanations. MemoryVLA (ICLR 2026) provides a relevant alternative based on perceptual and semantic memory [R1], while VaM adopts temporal attention from MEM [R2]. A comparison should hold the backbone and observation coverage fixed while varying what is cached…..
Theorem 1 (page 4, Eqs. 5-7) is a valid conditional distance argument, but any suitably separating added feature can relax its bound. It does not establish that the learned values provide the necessary separation…
Likewise, swapping an earlier critic into a trained actor at inference (page 7, Tab. 1) changes the actor’s input distribution. The performance drop could reflect value accuracy, actor-critic co-adaptation, or both. This distinction matters because value memory is the clearest contribution beyond existing memory and post-training systems.
The online comparison also needs clearer accounting. Eight times as many policy iterations at a fixed gradient-step budget is a scheduling choice, and lower time per gradient step can follow from concurrent collection and training. The success gain under the stated equal collection budgets is the stronger evidence, but neither those budgets nor the hardware and timing boundaries can be audited here.
Offline/online data mixing already appears in RLPD [R3], online robot adaptation in ConRFT and SERL [R4, R5], and heterogeneous experience with advantage conditioning in RECAP [R6]. The paper scheduling contribution needs to be separated from computational parallelism and evaluated against cumulative robot interactions and elapsed time.
References
[R1] Shi et al. MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation. ICLR 2026.
[R2] Torne et al. MEM: Multi-Scale Embodied Memory for Vision Language Action Models. arXiv:2603.03596.
[R3] Ball et al. Efficient Online Reinforcement Learning with Offline Data. ICML 2023
[R4] Chen et al. ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy. RSS 2025
[R5] Luo et al. SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning. ICRA 2024
[R6] Physical Intelligence et al. $\pi^{*}$0.6: A VLA That Learns From Experience. arXiv:2511.14759
Questions for authors
-
What raw-observation horizons do the actor, critic, and earliest cache entry cover? Does the no-cache condition retain the same proprioceptive encoder and comparable parameter capacity?
-
With the same backbone, observation coverage, and training budget, how does value caching compare with caching visual features or another compact progress representation? Can the critic-snapshot result distinguish value accuracy from the actor’s adaptation to that critic?
-
Please provide the referenced experimental appendices, evaluation episode and recovery-event counts, independent training-run counts, and the per-method demonstration, rollout, and intervention budgets. What uncertainty remains beyond variation among three checkpoints?
-
For Sec. 4.4, what are the collection and update budgets per iteration, hardware allocations, and operations included in elapsed time? How do the learning curves compare against total robot interactions and wall-clock time with a tuned iterated-offline schedule?
3 comments
Reviewer: please reply
Please reply to the authors, and indicate whether your concerns were met.
Dear Reviewer QayW,
Thank you again for taking the time to review our paper and share your suggestions. In our rebuttal, we clarify the observation horizons and experimental budgets, explain why we use value predictions as a compact representation of task progress, and report a visual-caching baseline and results from three independent training runs on Laundry Folding. Our cross-seed experiment suggests that actor–critic co-adaptation plays a larger role in the gains from value caching than differences in value-estimation accuracy. We also provide learning curves indexed by wall-clock time and cumulative training samples.
We would be grateful if you could review our updated responses and let us know whether they address your concerns. We welcome any further constructive feedback and would be happy to discuss any remaining questions during the discussion period. If the revisions have resolved your concerns, we would appreciate it if you would consider updating your assessment.
Best regards, Authors
We would greatly appreciate it if you could take a look at our first-round revisions. Please let us know if you have any further comments or suggestions. We would be happy to make additional improvements and engage in further discussion.