acceptodds
Under review as a conference paper at ICLR 2027

Value as Memory: Online Value-Guided Post-training for General Robot Policies

18 views

Abstract

Real-world post-training of generalist robot policies faces two challenges: partial observability makes action selection ambiguous, while offline post-training methods delay feedback from deployment experience. We present Value as Memory (VaM), a framework that addresses these challenges through value caching and online real-world reinforcement learning (RL). Value caching conditions actions on recent value predictions as task-progress memory, while online updates enable timely policy improvement by combining offline demonstrations, off-policy interventions, and on-policy rollouts for sample-efficient adaptation. As a result, VaM transitions from offline RL warmup to online autonomous learning with sparse rewards. Across long-horizon tasks involving deformable objects and contact-rich manipulation, VaM yields a mean absolute success-rate gain of 29.4% with value caching, while achieving a 32.8% gain with online autonomous learning over matched offline training. The evaluated online schedule performs as many policy iterations under the same gradient-step budget, with more than lower recorded time per gradient step with concurrent data collection and training.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Manipulation, Online Post-training, Real-world Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QayWOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper conditions a pretrained robot policy on recent scalar value predictions and adapts it through online reinforcement learning. The value history is encoded with proprioceptive history and supplied to a flow-matching policy initialized from $\pi$0.5. The method training proceeds via demonstration initialization, offline-RL, human-assisted collection, and autonomous learning that retains earlier interventions in replay. The paper’s idea is to use predicted values as policy memory. Experiments on three challenging physical tasks is presented to support the claim made in the paper.

Critical strengths

The central idea is appealing. The asynchronous implementation gives the idea practical usage. The action policy consumes previous value predictions while value inference proceeds separately, making the cache usable without requiring a current critic prediction before every action.

The strongest evidence is the consistency of the physical-task gains.

Improvements across deformable-object handling and contact-sensitive assembly is good.

The warmup ablation, earlier-critic substitutions and blind recovery annotation also provide more insight than final success alone.

Critical weaknesses

Removing the cache causes a large performance drop, so the main ablation does establish that the added input helps. What remains unclear is why it helps. Each cached prediction is computed from an observation history…. the oldest cached prediction may therefore expose the actor to information beyond the no-cache actor’s effective observation horizon. The paper does not specify those horizons….

Additional temporal context, learned features, and value-specific information remain plausible explanations. MemoryVLA (ICLR 2026) provides a relevant alternative based on perceptual and semantic memory [R1], while VaM adopts temporal attention from MEM [R2]. A comparison should hold the backbone and observation coverage fixed while varying what is cached…..

Theorem 1 (page 4, Eqs. 5-7) is a valid conditional distance argument, but any suitably separating added feature can relax its bound. It does not establish that the learned values provide the necessary separation…

Likewise, swapping an earlier critic into a trained actor at inference (page 7, Tab. 1) changes the actor’s input distribution. The performance drop could reflect value accuracy, actor-critic co-adaptation, or both. This distinction matters because value memory is the clearest contribution beyond existing memory and post-training systems.

The online comparison also needs clearer accounting. Eight times as many policy iterations at a fixed gradient-step budget is a scheduling choice, and lower time per gradient step can follow from concurrent collection and training. The success gain under the stated equal collection budgets is the stronger evidence, but neither those budgets nor the hardware and timing boundaries can be audited here.

Offline/online data mixing already appears in RLPD [R3], online robot adaptation in ConRFT and SERL [R4, R5], and heterogeneous experience with advantage conditioning in RECAP [R6]. The paper scheduling contribution needs to be separated from computational parallelism and evaluated against cumulative robot interactions and elapsed time.

References

[R1] Shi et al. MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation. ICLR 2026.

[R2] Torne et al. MEM: Multi-Scale Embodied Memory for Vision Language Action Models. arXiv:2603.03596.

[R3] Ball et al. Efficient Online Reinforcement Learning with Offline Data. ICML 2023

[R4] Chen et al. ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy. RSS 2025

[R5] Luo et al. SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning. ICRA 2024

[R6] Physical Intelligence et al. $\pi^{*}$0.6: A VLA That Learns From Experience. arXiv:2511.14759

Questions for authors
  1. What raw-observation horizons do the actor, critic, and earliest cache entry cover? Does the no-cache condition retain the same proprioceptive encoder and comparable parameter capacity?

  2. With the same backbone, observation coverage, and training budget, how does value caching compare with caching visual features or another compact progress representation? Can the critic-snapshot result distinguish value accuracy from the actor’s adaptation to that critic?

  3. Please provide the referenced experimental appendices, evaluation episode and recovery-event counts, independent training-run counts, and the per-method demonstration, rollout, and intervention budgets. What uncertainty remains beyond variation among three checkpoints?

  4. For Sec. 4.4, what are the collection and update budgets per iteration, hardware allocations, and operations included in elapsed time? How do the learning curves compare against total robot interactions and wall-clock time with a tuned iterated-offline schedule?

3 comments

We would greatly appreciate it if you could take a look at our first-round revisions. Please let us know if you have any further comments or suggestions. We would be happy to make additional improvements and engage in further discussion.

Area_Chair_ugruOpenReview ↗

Reviewer: please reply

Please reply to the authors, and indicate whether your concerns were met.

Dear Reviewer QayW,

Thank you again for taking the time to review our paper and share your suggestions. In our rebuttal, we clarify the observation horizons and experimental budgets, explain why we use value predictions as a compact representation of task progress, and report a visual-caching baseline and results from three independent training runs on Laundry Folding. Our cross-seed experiment suggests that actor–critic co-adaptation plays a larger role in the gains from value caching than differences in value-estimation accuracy. We also provide learning curves indexed by wall-clock time and cumulative training samples.

We would be grateful if you could review our updated responses and let us know whether they address your concerns. We welcome any further constructive feedback and would be happy to discuss any remaining questions during the discussion period. If the revisions have resolved your concerns, we would appreciate it if you would consider updating your assessment.

Best regards, Authors

Reviewer_WBdsOpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VaM conditions a generalist robot policy on recent value predictions and combines demonstration initialization, offline RL warmup, human-assisted online learning, and autonomous learning. On three real-world manipulation tasks, the paper reports substantial improvements from value caching and online updates. A local Lipschitz analysis motivates how additional cache features can relax an action-prediction error lower bound.

Critical strengths
  1. The three real-world tasks involve challenging deformable-object and contact-rich manipulation. The reported 29.4-percentage-point improvement over the no-cache variant is substantial, and the recovery analysis includes blinded annotation and raw event counts.

  2. Reusing a learned value signal as a compact policy input is a practical idea. The integration with continued autonomous learning is useful, and the paper provides explicit training objectives, data mixtures, and inference-time value-replacement experiments.

Critical weaknesses
  1. Using progress or value estimates to represent task progress is not new to me. [1] stores progress estimates and feeds progress differences into action selection, [2] conditions control on current and past spatial-progress maps. VaM uses an online TD-value cache, its contribution over these approaches needs clearer justification: the experiments do not isolate why temporal value history is preferable to a latest-value-only input or a conventional progress-history input. Replacing the critic at inference may also introduce input-distribution mismatch.

  2. [3] enables continuous data collection and asynchronous RECAP updates. Thus, comparison against sequential Offline RL does not establish a distinctive online-learning contribution. A resource-matched online baseline is needed. The per-stage schedule and hardware allocation are underspecified, and checkpoint-level standard deviations do not establish training-run robustness.

References:

[1] The Regretful Agent: Heuristic-Aided Navigation through Progress Estimation. CVPR, 2019.

[2] Mash, Spread, Slice! Learning to Manipulate Object States via Visual Spatial Progress. ICRA, 2026.

[3] SOP: A Scalable Online Post-Training System for Vision-Language-Action Models. Preprint, 2026.

Questions for authors
  1. Under matched architecture, observation history, and training budgets, does VaM outperform retrained controls using only the latest value or a conventional progress-history signal? These would clarify the specific benefit of value memory.

  2. What distinguishes the online update scheme from SOP-style online RECAP [3]? Please clarify the per-stage collection/update schedules, hardware allocation, and number of independent training runs supporting the comparison.

13 comments

We would greatly appreciate it if you could take a look at our first-round revisions. Please let us know if you have any further comments or suggestions. We would be happy to make additional improvements and engage in further discussion.

Thanks for clarifying that the online RECAP baseline uses the same asynchronous updates as VaM. The SOP comparison on Box Assembly addresses much of my concern about the online baseline. VaM and RECAP differ in both value caching and offline RL warmup, though, so it’s still hard to tell how much each contributes to the gain.

My main question is how much the earlier value predictions help once the policy has the latest one. The visual-cache comparison changes what goes into the cache, and the demonstration-only critic changes how the values are learned. Both results could hold even if the policy relied mostly on the latest value.

Could you please report results from a public long-horizon task? A composite task in RoboCasa, a multi-step RLBench task, a CALVIN instruction sequence, or a LIBERO task would all work. Could you train two policies on the same offline data, with the same fixed critic, encoder architecture, and training budget? One would get the full value history. The other would get the latest value repeated in every cache slot during both training and evaluation. Comparing their success rates from the same initial states would answer my question for a fixed critic. You can also upload the training and evaluation code to the supplementary materials so others can reproduce the comparison.

The three-seed results on Laundry Folding give me more confidence in that comparison. Did the additional runs start from a shared checkpoint, and did each seed collect its own robot rollouts?

For now, I’m keeping my current rating.

Thank you for the continued engagement and the precise experimental specification. We selected the long-horizon composite task RecycleBottlesByType from RoboCasa365 for this comparison. We have added reproducible RoboCasa simulation training and evaluation code to the supplementary material and included this comparison experiment in Appendix H of the revised paper.

On whether per-frame value history helps beyond the latest value

We ran the requested comparison on a RoboCasa long-horizon composite task: two policies trained on the same offline expert data, same encoder architecture and training budget, evaluated on the same 100 scenarios per cache variant and seed across three seeds (42, 44, and 46). The only difference is the cache input — constant-latest-value (latest scalar replicated across all slots) vs. per-frame value history (actual sequence of recent predictions).

Success rates are reported as mean ± sample standard deviation across the three seeds. For each cache variant and seed, we trained offline on 520 expert demonstration episodes for 20,000 steps, collected 100 rollout episodes, and then trained the value and action experts for 5,000 steps each using a 1:1 expert-to-rollout sample ratio (24 expert and 24 rollout samples per batch). The value expert is trained with Monte Carlo (MC) targets during the offline stage and temporal-difference (TD) targets during the online round.

Value cacheSeed 42Seed 44Seed 46Mean ± SD (%)
Offline only
Constant74%74%75%74.3 ± 0.6
Per-frame66%68%78%70.7 ± 6.4
After one online round
Constant65%69%68%67.3 ± 2.1
Per-frame69%67%75%70.3 ± 4.2

The offline stage matches the reviewer's "fixed critic" condition — the critic is trained purely on offline demonstrations. After one round of on-policy rollouts with TD value adaptation, advantage labeling, and action training, the mean gap reverses: constant leads by 3.7% offline, while per-frame leads by 3.0% online.

We believe this pattern is explained by [1], whose partial-observability setting parallels our soft-aliasing problem (Sec. 3.2.2). They identify a latching effect: learners trained only on expert data treat their own actions as expert-generated, prematurely collapsing the context posterior. Their experiments show that access to history worsens off-policy performance.

In our offline stage, the expert-trained critic encodes expert-specific progress curves. Per-frame history allows the policy to latch onto these temporal correlations, while the less expressive constant cache offers fewer such shortcuts and generalizes better offline.

Adding on-policy rollouts reverses this pattern, as predicted by Theorem 5.4 in [1]. In our setting, TD updates calibrate the critic to the learner's distribution, making per-frame history useful for disambiguation (Sec. 3.2.2): it indicates improving, stalling, or declining progress, while the constant cache discards this temporal profile.

On the three-seed Laundry Folding runs

Each seed was fully independent, including the offline training stage. Therefore, each seed collect its own robot rollouts.

[1] Swamy, Gokul, et al. "Sequence model imitation learning with unobserved contexts." Advances in Neural Information Processing Systems 35 (2022): 17665-17676.

Reviewer_Q9dSOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduce VaM as a offline-to-online RL framework for post-training robot policies. The idea is to feed recent value predictions to the action policy as a kind of task-progress memory. The policy and the value model are then updated using demonstrations, human-intervention data, and autonomous rollouts. On three real-world manipulation tasks, they report a 29.4 point average success-rate gain over the variant without value caching, and a 32.8 point gain for online learning over matched offline training.

Critical strengths
  1. The idea of using recent value estimates as policy input is interesting. It’s a pretty direct way to represent task progress in cases where the visual observation alone is ambiguous.
  2. The tasks are hard and physical, including deformable-object manipulation and contact-rich assembly, so the paper isn’t leaning only on simulation benchmarks.
  3. The online training design is practical. It reuses the offline demonstrations and intervention data together with new rollouts, and collecting data while training also gives more frequent policy iterations under the reported gradient-step budget.
Critical weaknesses
  1. The uncertainty numbers is hard to interpret. Table 1 give the mean and standard deviation over the final three checkpoints. That show how much the checkpoints vary, but it doesn’t tell me how much results vary across independent training runs.
  2. The role of the progress signal could be isolated better. The no-cache ablation supports the value-caching design, but it doesn’t rule out that the gains just come from having extra inputs or more model capacity. A parameter-matched control with shuffled, constant, or delayed value histories would show whether the progress information is what actually matters.
  3. The real-world evaluation is narrow. It covers three tasks on one robot, and online DAgger uses human interventions before the autonomous learning starts. So it can’t say whether this transfers to other robots or a wider set of tasks.
Questions for authors
  1. Could you add a parameter-matched control using shuffled or constant value histories, to test whether the temporal progress information in the cache is what drives the improvement?
  2. Are the reported variations based only on checkpoints, or do they include independent training runs? Could you report run-level variability and the number of evaluation episodes?
2 comments

We would greatly appreciate it if you could take a look at our first-round revisions. Please let us know if you have any further comments or suggestions. We would be happy to make additional improvements and engage in further discussion.

Dear Reviewer Q9dS,

Thank you again for your thoughtful feedback. In our updated responses, we report results from three independent training runs on Laundry Folding, where VaM consistently outperforms the ablations and the RECAP baseline. On Box Assembly, VaM also outperforms a parameter-matched baseline that caches values learned only from successful demonstrations, suggesting that the quality of the cached progress signal matters. We also clarify the scope of our real-world evaluation and point to the reported evaluation episode counts.

We would be grateful if you could review our updated responses and let us know whether they address your concerns. We welcome any further constructive feedback and would be happy to discuss any remaining questions. If the revisions have resolved your concerns, we would appreciate it if you would consider updating your assessment.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.