acceptodds
Under review as a conference paper at ICLR 2027

Online Robot Policy Refinement via Dynamics-Value Decomposition

Abstract

Pretrained imitation policies can solve complex manipulation tasks, yet small errors can still cause failure in precise manipulation. Online reinforcement learning (RL) can correct such failures, but most policy refinement methods rely on -functions whose offline initialization can be unreliable for the nearby unseen actions introduced during refinement. Rather than directly pretraining a -function, we compose a pretrained latent dynamics model, adapted with task-specific offline transitions, with a state-value model initialized through task-progress supervision. We call this dynamics-value decomposition: an action chunk is evaluated by predicting its latent successor and scoring the predicted state. This composition provides an initial action evaluator that generalizes locally to action perturbations during refinement. During online RL, new transitions update the dynamics model, TD learning calibrates the value model to environment returns, and policy gradients propagate through their composition. Instantiated with DINO-WM and RoboMeter, our action evaluator replaces direct -functions in residual and latent-steering policy refinement methods. Across precise manipulation tasks in simulation and the real world, our method improves online sample efficiency over the corresponding -based methods.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Reinforcement Learning, Robotic Manipulation, Online Finetuning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1uKUOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes replacing the direct Q-function in online policy-refinement methods with a composed action evaluator, S_DV(s, a) = V(f(e(s), a)). Here f is a one-chunk latent dynamics model built on DINO-WM / frozen DINOv2 features and adapted on base-policy rollouts, and V is a state-value ensemble that is first distilled from RoboMeter task-progress estimates and then trained with TD online. The actor maximizes γ^H S_DV by backpropagating through both components, and S_DV also ranks candidate action chunks. The evaluator is plugged into DICE-RL (residual actions) and DSRL (latent-noise steering) and evaluated on four MimicGen settings (3 seeds) and two real-robot insertion tasks (DICE-RL only). The authors report gains over the corresponding direct-Q variants on three of the four simulated settings and on one of the two real tasks.

Critical strengths
  • The motivation is clear. Separating "where does this action lead" from "how good is the resulting state" is intuitive, and each sub-question admits a type of supervision that a monolithic Q-function cannot directly use.
  • The evaluator is a drop-in replacement and is shown with two structurally different refinement methods (residual and latent steering).
  • Real-robot experiments are included, and the negative result on Threading is reported and discussed honestly.
  • The attention-readout visualization (Appendix B) is a useful qualitative look at what progress distillation changes in the value representation.
Critical weaknesses

W1. The claimed mechanism is not isolated, and the comparison is not controlled (main concern)

The paper's central claim is that decomposing the evaluator into dynamics and state value generalizes better to nearby unseen actions than a direct Q-function. Since V∘f and Q are equivalent in what they can represent, any advantage must come from either (a) the structure itself (e.g., dense next-state targets for f; V reusing states reached via other trajectories), or (b) access to pretrained components and extra supervision that the baseline does not receive. The experiments do not distinguish these.

(a) The comparison is not controlled. Sec. 4.1 states that the comparison "isolates the effect of replacing direct action-value estimation with dynamics–value decomposition." Tables 2–3 contradict this. Besides the evaluator, the two arms differ in:

  • visual representation: IMPALA trained from scratch vs. frozen DINOv2 with a learned readout;
  • critic initialization: random vs. RoboMeter progress for DICE-RL;
  • batch size (256 vs. 64), update frequency, gradient steps per update, target update rate, ensemble size and aggregation (10 heads/min vs. 3 nets/mean), DrQ augmentation, and replay size (DSRL);
  • training budget: Threading/DICE-RL uses 600k steps for Q vs. 300k for the proposed method, which also makes "final-checkpoint" comparisons ambiguous;
  • task-specific tweaks applied only to the proposed method: online progress distillation for Square D0 only, and segment-level readout pretraining for Square D1 and Threading (DSRL).

(b) Supervision is unequal, all in favor of the proposed method. The proposed method receives RoboMeter progress labels, pretrained DINOv2 features, next-state prediction targets, and, in the real-world experiments, additional expert demonstrations for progress distillation. The DICE-RL direct-Q baseline receives only sparse TD targets on a randomly initialized critic. So it is not shown that, under equal supervision, the decomposed evaluator generalizes better.

(c) There is no direct evidence for the generalization claim. No experiment measures how accurately either evaluator scores nearby unseen actions, which is the core claim illustrated in Fig. 1. Appendix B compares two state-value targets (progress vs. MC returns), not decomposed vs. direct evaluators. If anything, it shows that denser labels help regardless of decomposition, which reinforces the confound.

(d) Plausible alternative explanations are not ruled out:

  • pretrained frozen features vs. a critic encoder learned from sparse reward;
  • value-guided best-of-16 selection with a progress-aware scorer from the very first episode, while the baseline ranks candidates with a randomly initialized Q-function;
  • training stability: the V-TD target involves no maximization over actions, which may avoid Q-overestimation independent of any generalization argument.

I ask the authors to either (i) provide controlled comparisons with matched hyperparameters and budgets, including:

  • a direct Q-function on the same frozen DINOv2 features and readout, regressed to RoboMeter progress of s_{t+H} (and optionally with a next-state auxiliary loss);
  • the proposed method with randomly initialized V (no RoboMeter);
  • the base policy with best-of-16 selection by S_DV and no RL updates;
  • an offline evaluation of ranking/prediction accuracy of each evaluator on perturbed actions a + δ against held-out rollouts;

or (ii) reframe the contribution without the generalization claim, e.g., as a practical way to use pretrained world and progress models in policy refinement. Even under (ii), the hyperparameter and budget differences need to be removed.

W2. The generalization mechanism is underdeveloped

The argument for why the decomposition should generalize better appears only in passing. Sec. 3.2 notes that "every rollout transition provides a target for the dynamics model," and Sec. 3.3 that "both successful and failed rollouts provide dense supervision." But neither is connected to the generalization claim, which is stated as "local compositional generalization" without explanation. The density argument for V rests on RoboMeter labels rather than on the decomposition. Moreover, dense next-state supervision does not guarantee that f is sensitive to the action. With offline data from a single base policy, action variety per state is limited, and f could predict s' mostly from s while responding weakly or incorrectly to a. Policy refinement depends exactly on ∂f/∂a, yet no measure of action sensitivity or counterfactual prediction accuracy is reported.

W3. Technical soundness of the evaluator

  • Dropping the reward/termination term (Eq. 4 → Eq. 3). The justification is that non-terminal chunks dominate. However, the chunk in which success occurs is precisely where the precise corrections targeted by the paper matter. On that chunk, V is evaluated on a predicted terminal state. TD never trains V on terminal states as inputs, so V there is anchored only by the initial progress distillation (online distillation is used only on Square D0) and may drift.
  • What V estimates. V is trained by TD on replay data mixing offline base-policy rollouts and older online policies, with no action-dependent correction. It therefore estimates the value of the data-collecting policies, not the current or optimal policy, making the method closer to one-step policy improvement than to Q-learning. This should be discussed.
  • Bounding the correction and model exploitation. Eq. 13 states that α keeps candidate actions within the region covered by the dynamics model, but Table 2 lists "residual scale 1, no norm cap" with only an L2 penalty. Nothing explicit prevents the actor from exploiting errors in f.
  • Time-limit truncation is treated as terminal without bootstrapping (Tables 2–3). This applies to both arms, but it biases value targets.

W5. Novelty

Evaluating actions as V(f(s, a)) and backpropagating through a learned model is established (SVG(1)/value gradients, IVG, Dreamer, TD-MPC, model-based value expansion). RISE also combines a world model with a progress value model. The paper acknowledges this and positions its contribution as component-wise initialization for policy refinement. That places the burden on rigorous empirical evidence, which W1–W4 find lacking.

Questions for authors
  1. Which parts of f are actually pretrained? Sec. 3.3 says only the prediction head of a pretrained DINO-WM is fine-tuned, but Appendix A.2 describes substantial architectural changes (1-D conv action encoder, learnable query tokens, proprioception head), and Fig. 5 varies "DINO-WM training steps" from 50k to 300k. Is the predictor trained from scratch on D_off over frozen DINOv2 features?
  2. Does the DSRL direct-Q baseline also receive RoboMeter initialization? Table 3 lists "RoboMeter progress (offline)" without distinguishing the two arms.
3 comments

Thank you for the detailed review and the concrete controls you proposed. Below, DVD is our critic V(f(s,a))V(f(s,a)), composed of a dynamics model (ff) and a state-value model (VV) initialized from RoboMeter task-progress labels, and QQ is the original critic of DICE-RL or DSRL, which refine a pretrained base policy.

 

[W1] The claimed mechanism is not isolated.

We ran the four controls you requested and an equal-budget comparison in W1(a). The gain largely survives the controls below, and the ranking test on shifted actions favors DVD, as shown in W1(c).

  • DICE-RL+Q on frozen pretrained DINOv2 features. We replaced the IMPALA encoder of DICE-RL+Q with the frozen DINOv2 features and readout that DVD uses. On Square D0, it reaches 77%, against 78% for IMPALA-based Q and 88% for DICE-RL+DVD (200k to 300k steps). Pretrained features alone do not explain the gain.
  • The same critic, regressed to RoboMeter progress. Initializing this critic by regression to RoboMeter progress, as you proposed, still gives 77%, 11 points below DVD. Giving Q both pretrained components does not close the gap.
  • DVD with a randomly initialized value model. Without RoboMeter initialization, DVD loses at most 4.4 points on 6 of the 8 method-task pairs and still outperforms Q wherever the full DVD does. RoboMeter initialization is not the main source of the gain.
  • Best of 16 with the composed evaluator, no RL. Before any online training, executing the base-policy sample that the composed evaluator ranks highest out of 16 changes success by only +1.5 points over the base policy on average. DVD's gain comes mainly from online training.

Together, these controls support our contribution: DVD’s gain does not come from its pretrained components alone, but from replacing the direct Q critic with a decomposed evaluator trained online. Trained online, DVD improves DICE-RL and DSRL, and its critic ranks nearby unseen actions more accurately than Q (Section 4.3, Figure 5a).

 

[W1(a)] The comparison is not controlled.

We agree that the submission’s training budgets differed between variants. All main variants now train for 600k environment steps without online distillation, which had no significant effect, and we report the mean of the last three evaluations. This removes the budget mismatch and the final-checkpoint ambiguity. DVD outperforms Q by 11 to 18 points on six of the eight method-task pairs (two RL methods on four tasks; revised Figure 3). On Threading, DICE-RL+Q is 10.7 points better, and the DSRL variants tie.

When we rerun DSRL+Q on ManiSkill3 LiftPegUpright with DVD’s batch size, update cadence, critic count and replay size, it reaches 82.0% from ID and 11.5% from OOD initial states (80.4% and 10.3% with its default settings, Table 1), against 88.7% and 46.3% for DSRL+DVD. On MimicGen, hyperparameters remain only partly matched (Tables 3 and 4): DVD runs its dynamics model in every update, which limits its batch size to 64 on the same GPU (256 for Q), and the two also differ in update cadence, target update rate, critic ensemble, DrQ augmentation and, in DSRL, replay size and readout pretraining. Q keeps the published DICE-RL and DSRL recipes, while DVD’s settings follow from its cost and design: its dynamics model runs in every update, which also limits its update cadence, and its value model reads frozen DINOv2 features, so there is no critic encoder to augment.

[W1(b)] Supervision is unequal.

We agree. Our controls give Q the DINOv2 features and RoboMeter initialization of DVD, which does not help Q, and remove RoboMeter from DVD, which preserves most of its gain (W1). We have not run DVD without its DINOv2 features, and on the real robot its value model also receives expert demonstrations. We therefore view the real-robot comparison as a feasibility result under unequal supervision rather than a controlled comparison.

 

[W1(c)] Direct evidence for generalization.

We ran this test and added it to the revised paper (Section 4.3, Figure 5a).

At the last DSRL checkpoint (600k), the composed evaluator ranks nearby unseen actions correctly with an accuracy of 0.60, against 0.49 for Q, which is at chance. Its lead of 0.11 (95% CI 0.07 to 0.15) holds on all four tasks, significantly on three of them. From held-out states, we shift the executed 8-action chunk along each of the six end-effector axes in turn (x, y, z and three rotations) to nine candidates within ±0.4 and score each by its true return from 16 rollouts that use the same random seeds for every candidate.

[W1(d)] Alternative explanations.

Our results argue against the first two explanations and only partly against the third. Frozen features do not help Q (within 3 points of IMPALA-based Q on DICE-RL Square D0 and D1). Selection without RL nearly matches the trained gain only on HammerCleanup (+12 versus +13 points), and there DICE-RL+DVD with a randomly initialized V, which starts without a progress-aware scorer, matches the full DVD. DSRL, which performs no selection, also gains from DVD.

On the third, Q does not overestimate: taking the minimum over its heads, it underestimates realized returns by 0.01 to 0.11. This pessimism may still slow Q independently of generalization, which our controls do not isolate.

 

[W2] Underdeveloped generalization mechanism.

Q learns an action's effect only from sparse TD targets on executed actions, whereas the dynamics model learns it from dense next-state targets on every transition, and the value model can score a predicted state that other trajectories have reached. The density argument does not rest on RoboMeter alone: with a randomly initialized V, DICE-RL+DVD and DSRL+DVD retain most of the gain.

 

[W3.1] Dropping the reward/termination term.

Dropping the term rarely changes the chosen chunk (DICE-RL, Square D0). On the chunk that completes the task, DVD scores the chosen chunk at 0.88 instead of 1, but in 94-97% of these decisions all 16 candidates succeed, so the choice does not change, and the value model does not drift on real terminal frames. One decision earlier, DVD skips a chunk that would complete the task immediately in 7% of successful episodes (1-5% for DICE-RL+Q), which delays success but does not prevent it.

 

[W3.2–W3.4] Value target, bounding the correction and time-limit truncation.

The value model indeed estimates the value of the data-collecting policies, so the method is closer to one-step policy improvement, and our comparison is between one-step policy improvement and off-policy Q-learning. In practice, against the evaluated policy's return, its bias stays within -0.03 to +0.02 in DSRL and -0.06 to 0.00 in DICE-RL.

The text at Equation (13) overstates how the correction is bounded. With residual scale 1 and no norm cap, only the L2 penalty and best-of-16 selection bound the correction. We have not directly tested whether the actor exploits errors in the dynamics model.

Treating time-limit truncation as terminal biases both critics' targets near the time limit in the same way, since both critics use this treatment.

 

[W5] Novelty: V(f(s,a)) and model-based value gradients are established.

We revised Section 2 (L124-L147) to state that DVD builds on the value-gradient mechanism of SVG and IVG. Our contribution is using it as a drop-in critic that improves existing online refinement methods for frozen pretrained policies.

IVG [1] is the closest work. It learns its policy, value and reward from scratch, mostly with shaped rewards, and its IVG(1) variant scores actions as V(f(s,a)) plus a learned reward. DVD scores actions like IVG(1) without the reward term, to refine a frozen pretrained policy online under sparse reward. RISE [2] keeps its world model and value model frozen and does not interact online, whereas DVD trains both online. Within 600k steps, DICE-RL+DVD outperforms DICE-RL+Q by 11.3 points on Square D0.

 

[Q1–Q2] Pretrained parts of f, and RoboMeter initialization for DSRL+Q.

In the simulated experiments, only the frozen DINOv2 encoder is pretrained. There, the predictor is trained from scratch on base-policy rollouts, then fine-tuned online on replay that includes demonstrations. "DINO-WM" names only the architecture family; apart from the frozen DINOv2 encoder, the dynamics model is not pretrained.

Neither Q baseline, including DSRL+Q, receives RoboMeter initialization: both start from random weights.

 

References

[1] Byravan, A., et al. (2020). Imagined value gradients: Model-based policy optimization with transferable latent dynamics models. CoRL 2020.

[2] Yang, J., et al. (2026). RISE: Self-improving robot policy with compositional world model. arXiv:2602.11075.

Dear Reviewer 1uKU,

Thank you again for your detailed feedback. If you have a chance to read our rebuttal, we’d appreciate hearing whether it addresses your concerns. We’d be happy to clarify any points that remain unclear.

Reviewer_aXuzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method of decomposing a Q-function into a state-value function and a latent dynamics model. Both two models are initialized with pre-trained models and then finetuned. Within two RL framework, the proposed method demonstrates performance gain over variant of directly learning a Q-function. The method is evaluated in both simulation and real-wolrd manipulation tasks.

Critical strengths
  1. The idea is clear and the method demonstrate performance gain on existing methods.
Critical weaknesses
  1. The evaluation is limited and it does not explicitly show the performance gain from the architecture design. The performance gain could mainly comes from leveraging the pre-trained models.
Questions for authors
  1. Could you run some experiments to demonstrate the generalization ability of the dynamics model and the state-value model on OOD-actions? Otherwise, I did not understand why the proposed method could have strong performance gain compared to finetuning a Q-function directly on a pre-trained model.
  2. Have the authors tried to use a longer action chunk in the Q-estimation using the latent dynamics model and the state-value function? Is there a balance between target value accuracy and stability like the difference between Monte-carlo target and TD-target?
3 comments

Thank you for noting that the idea is clear and that the method improves the performance of existing methods. We added the equal-budget, new-task, unseen-action and chunk-length results below to the revised Section 4.3.

 

[W1] "The evaluation is limited", and the gain may come "from leveraging the pre-trained models".

We agree that the pretrained components (the RoboMeter-initialized value model and the DINOv2 encoder) could explain the gain. Therefore, we additionally conducted the ablation studies below.

  • DVD without RoboMeter initialization. We replaced the RoboMeter initialization of the value model with a random one. DVD then loses at most 4.4 points on 6 of the 8 method-task pairs (DICE-RL and DSRL on four MimicGen tasks), but still outperforms Q wherever the full DVD does. RoboMeter initialization is not the main source of the gain.
  • Q with the DINOv2 encoder and RoboMeter initialization. We gave a DICE-RL Q critic the frozen DINOv2 features that DVD uses, with or without RoboMeter initialization. It performs at the level of IMPALA-based Q, 11 points below DVD on Square D0 (200-300k steps). The pretrained components alone do not explain the gain.
  • Selection without RL. Choosing the best of 16 base-policy samples with the composed evaluator V(f(s,a))V(f(s,a)) before online training changes success by only +1.5 points over the base policy on average, against +20.5 for trained DICE-RL+DVD; only on HammerCleanup is it comparable (+12 versus +13 points). DVD's gain comes mainly from online training.

At an equal budget of 600k online environment steps, DVD outperforms Q by 11 to 18 percentage points in 6 of 8 method-task pairs (revised Figure 3). On Threading, DICE-RL+Q performs 10.7 points better, while the DSRL variants tie.

We also evaluated a new task, ManiSkill3 LiftPegUpright, now in the revised paper (Figure 3, Table 1). DVD’s gain is clearest from OOD initial states, where it raises DSRL’s success on LiftPegUpright from 10.3% to 46.3%. From ID initial states it leads or matches Q (DSRL: 80.4% to 88.7%).

 

[Q1] Generalization "on OOD-actions", and Q fine-tuned "on a pre-trained model".

Yes. On actions the data never contained, the value model applied to the dynamics model's prediction, V(f(s,a)), ranks candidates better than a Q critic, and the dynamics model's predicted end-effector movement tracks the true one (r = 0.97). We added the ranking test to the revised paper (Section 4.3, Figure 5a).

We shifted the executed 8-action chunk along each of the six end-effector axes in turn (x, y, z and three rotations) to nine values in [-0.4, +0.4] and measured each candidate's true return from 16 rollouts. DSRL+DVD's V(f(s,a)) orders candidate pairs correctly 60% of the time at the last checkpoint (600k), against 49% for the DSRL+Q critic, which is at chance, and it is ahead on all four tasks. The Q critic mostly extrapolates "push further": its score keeps rising or falling with the shift in 76% of sweeps, against 28% for V(f(s,a)) and only 4% for the true return.

Fine-tuning Q directly on the same pretrained features does not close the gap: DICE-RL+Q on frozen DINOv2 features stays 11 points below DVD on Square D0 (W1). A Q critic learns the effect of an action only from sparse returns, whereas the dynamics model learns it from a next-state target on every transition.

 

[Q2] Longer chunks, and "a balance between target value accuracy and stability".

We evaluated chunk lengths H∈{8,16,24}H \in \{8, 16, 24\}. Figure 5b reports all four tasks in the revised paper. Below are Square D0 results after 600k environment steps, with success rates (%) and Spearman value-return correlations:

Longer chunks reduce performance, including for the unchanged base policy, but DVD retains higher success and stronger value-return correlation than Q. Because chunk length also changes execution duration before replanning, this does not isolate a Monte Carlo–TD trade-off; nor does correlation alone establish training stability.

When matching the number of updates, DVD still retains its advantage at H = 24, achieving 78% versus 53% for Q.

Success (%)H = 8H = 16H = 24
Base policy745842
DICE-RL+Q81.061.040.3
DICE-RL+DVD (ours)92.385.863.3
Value-return correlationH = 8H = 16H = 24
DICE-RL+Q0.860.710.52
DICE-RL+DVD (ours)0.930.830.76

Thank you for your reply and additional experiments. They addressed most of my concern and misunderstanding. I would like to raise the score to 3.

Thank you again for your thoughtful feedback and for taking the time to review our response and additional experiments. We’re glad the new results helped address your concerns.

Reviewer_tWVnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an online policy refinement method by decomposing the dynamics and task values and separately finetunes the two components during online RL. The key idea is that instead of training and evaluating on a Q function directly, the proposed method evaluates the action chunk based on the scoring of the predicted future successor states. The empirical study compares the proposed method with existing Q-function based methods and demonstrates certain performance improvements in terms of success rates on both simulation and real-world experiments.

Critical strengths
  • Clear Presentation: The overall presentation of the proposed method is clear and easy to follow. The motivation of conducting dynamics-value decomposition seems to be clear.

  • Strong Empirical Performance: The proposed method demonstrate stronger performance compared to existing baselines such as DICE-RL and DSRL. The paper also provides both simulated and real results, supporting with an acceptable amount of ablation studies.

Critical weaknesses
  • Missing discussions and comparisons: The dynamics-value decomposition is not a new technique overall. Prior works [1,2] also employ this similar kind of decomposition. Please discuss the differentiation between the proposed method and these line of works. Also consider choosing them as baselines in a addition to existing Q-based methods.

  • Additional tasks and environments: The paper evaluates the proposed method mainly on mimicgen gripper manipulation tasks. More simulation results on more difficult tasks in ManiSkill3 [3] or Robotwin [4] may strengthen the contribution of the proposed method.

[1] Byravan, A., Springenberg, J. T., Abdolmaleki, A., Hafner, R., Neunert, M., Lampe, T., ... & Riedmiller, M. (2020, May). Imagined value gradients: Model-based policy optimization with tranferable latent dynamics models. In Conference on Robot Learning (pp. 566-589). PMLR.

[2] Yang, J., Lin, K., Li, J., Zhang, W., Lin, T., Wu, L., ... & Li, H. (2026). Rise: Self-improving robot policy with compositional world model. arXiv preprint arXiv:2602.11075.

[3] Tao, S., Xiang, F., Shukla, A., Qin, Y., Hinrichsen, X., Yuan, X., ... & Su, H. (2024). Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai. arXiv preprint arXiv:2410.00425.

[4] Chen, T., Chen, Z., Chen, B., Cai, Z., Liu, Y., Li, Z., ... & Mu, Y. (2025). Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506.18088.

Questions for authors

See the weakness section.

3 comments

Thank you for finding our dynamics-value decomposition (DVD) clearly presented and well motivated, with gains over DICE-RL and DSRL in simulation and, with DICE-RL, on real robots.

[W1-1] Differences from IVG and RISE, since the decomposition "is not a new technique overall".

We agree that composing a dynamics model (ff) with a state-value model (VV) and backpropagating values through learned dynamics are established ideas, including in SVG [1] and IVG [2]. Our contribution is not the decomposition alone, but its use as an online-adapted action-chunk evaluator for refining frozen pretrained policies. DVD replaces the original Q critic of DICE-RL and DSRL.

  • (L127-L133) IVG [2] optimizes a policy through imagined latent rollouts using predicted dense rewards and values. DVD instead uses a single chunk-level prediction and successor-state value, without a separate reward prediction to refine actions from a frozen pretrained policy online under sparse reward.
  • (L136-L145) RISE [3] improves a pretrained vision-language-action policy in imagination, without additional real-world interaction during self-improvement. Its dynamics and value models remain frozen, and the policy is updated through advantage-conditioned training, rather than computing policy gradients through the model. DVD instead trains its dynamics model and value model online from real interaction under sparse reward and takes the actor's gradient through V(f(s,a))V(f(s,a)).

The original submission compared these distinctions in Section 2, and we made this clearer in the revised paper (L124-L147).

 

[W1-2] IVG and RISE as baselines.

To investigate whether longer imagined rollouts, as in IVG, improve refinement within the same framework, we have compared with an IVG-inspired multi-step DVD. This variant computes the actor gradient from the discounted value of the state predicted two action chunks ahead, rather than one. On Square D0, its success rate is 1.5 percentage points below DICE-RL+DVD with one chunk. We therefore observed no improvement from the additional imagined chunk in this experiment.

We agree that RISE is a relevant model-based comparison. However, we have not added a full RISE implementation since our experiments focus on isolating the effect of replacing the Q critic within online-refinement frameworks (DICE-RL and DSRL). A full RISE comparison would additionally change the source of training experience, the model-adaptation protocol, and the policy-update mechanism, rather than isolate the action evaluator.

 

[W2] Harder tasks in ManiSkill3 or RoboTwin.

Following your suggestion, we added experiments on ManiSkill3 LiftPegUpright, expanding the evaluation beyond MimicGen. The results are included in the revised paper (Figure 3, Table 1). Due to limited time and compute during the rebuttal period, we could not include RoboTwin 2.0 experiments.

On ManiSkill3 LiftPegUpright with the DiT base policy, we evaluate from in-distribution (ID) initial states and from out-of-distribution (OOD) initial states, where the peg starts outside the range of the demonstrations. From ID initial states, the base policy already succeeds 71 to 78% of the time, which leaves little room for improvement: DSRL+DVD improves on DSRL+Q from 80.4% to 88.7%, and DICE-RL+DVD and DICE-RL+Q perform similarly (both above 90%). From OOD initial states, the base policy succeeds only 3.5% of the time, so improvement must come from online refinement, and the gain is clearer: DVD raises success from 10.3% to 46.3% with DSRL and from 80.3% to 84.0% with DICE-RL.

All rows are ManiSkill3 LiftPegUpright with the DiT base policy and 600k environment steps of online training. Each entry is the success rate (%) at the last checkpoint.

Task, method, initial statesBase policyQ criticDVD (ours)
LiftPegUpright, DSRL, ID78.080.488.7
LiftPegUpright, DSRL, OOD3.510.346.3
LiftPegUpright, DICE-RL, ID7194.795.3
LiftPegUpright, DICE-RL, OOD3.580.384.0

These additional experiments on ManiSkill3 provide evidence that DVD’s benefits extend beyond MimicGen.

 

References

[1] Heess, N., et al. (2015). Learning continuous control policies by stochastic value gradients. NeurIPS 2015.

[2] Byravan, A., et al. (2020). Imagined value gradients: Model-based policy optimization with transferable latent dynamics models. CoRL 2020.

[3] Yang, J., et al. (2026). RISE: Self-improving robot policy with compositional world model. arXiv:2602.11075.

Thanks for your detailed response. My concerns have been addressed mostly. I'll increase my score to 3.

Thank you again for raising this concern and for taking the time to consider our response. We’re glad our clarification was helpful, and we appreciate your thoughtful feedback.

Comments on the paper

1 comment

Dear Area Chair,

Thank you for overseeing the discussion. Reviewers tWVn and aXuz both raised their scores to 3 after considering our responses and additional experiments, explicitly stating that most of their concerns had been addressed. As the discussion closes, we would like to highlight the new evidence, particularly the controls and novelty requested in Reviewer 1uKU’s main concern.

 

(1) Controlled training budget [1uKU]

All methods now train for the same 600k environment steps. Under this equal budget, DVD's advantage over direct Q still holds and becomes clearer with longer training. These details are added to Section 4.3 and Figure 3.

 

(2) Accurate ranking nearby unseen action chunks [aXuz, 1uKU]

We perturbed action chunks from states reserved for evaluation and used Monte Carlo returns from executing those chunks to establish their ranking. DSRL+DVD ranked the perturbed action chunks more accurately than DSRL+Q on every tested MimicGen task. These details are in Section 4.3 and Figure 5a.

 

(3) Difference to prior work, IVG and RISE [tWVn, 1uKU]

DVD predicts the latent state reached by an action chunk and scores it with a state-value model. In Section 2 (Related Work), we now state that DVD builds on IVG's value gradients. DVD differs in setting: IVG learns policy, value and reward from scratch through multi-step imagined rollouts, whereas DVD uses a one-chunk prediction without a reward model as a drop-in critic that refines a frozen pretrained policy online under sparse reward.

We also explained in Section 2 that, unlike RISE, which keeps its world and value models frozen and improves the policy in imagination, DVD adapts both models from real online interaction under sparse rewards.

 

(4) Whether DVD's gains extend to ManiSkill3 [tWVn]

In addition to the MimicGen tasks, we evaluated DVD on ManiSkill3 LiftPegUpright. DVD's gains over direct Q were clearest when the peg started outside the demonstrated range. These details are in Section 4.3, Figure 3, and Table 1.

 

(5) Whether DVD's advantage holds across action-chunk lengths [aXuz]

We added a chunk-length ablation in Section 4.3 and Figure 5b. With DICE-RL, DVD yielded larger average success gains over the base policy than direct Q at each tested chunk length.

 

(6) Whether pretrained features and value initialization explain DVD's gains [aXuz, 1uKU]

In the rebuttal, we clarified that pretrained features alone do not explain DVD's gains. Even when direct Q used the same frozen pretrained visual features as DVD, with or without RoboMeter-based initialization, it still performed below DVD on the tasks we tested.

We also clarified that DVD's gain comes mainly from the decomposition itself rather than from RoboMeter initialization. DVD keeps most of its gain with a randomly initialized value model, and RoboMeter initialization improves it further on some tasks.

 

We appreciate Reviewer 1uKU’s thoughtful feedback and suggestions. Their current review predates these additional experiments, and we have not yet received a follow-up assessment. We hope the additional evidence and clarifications will help inform your final decision.

Thank you for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.