- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes replacing the direct Q-function in online policy-refinement methods with a composed action evaluator, S_DV(s, a) = V(f(e(s), a)). Here f is a one-chunk latent dynamics model built on DINO-WM / frozen DINOv2 features and adapted on base-policy rollouts, and V is a state-value ensemble that is first distilled from RoboMeter task-progress estimates and then trained with TD online. The actor maximizes γ^H S_DV by backpropagating through both components, and S_DV also ranks candidate action chunks. The evaluator is plugged into DICE-RL (residual actions) and DSRL (latent-noise steering) and evaluated on four MimicGen settings (3 seeds) and two real-robot insertion tasks (DICE-RL only). The authors report gains over the corresponding direct-Q variants on three of the four simulated settings and on one of the two real tasks.
Critical strengths
- The motivation is clear. Separating "where does this action lead" from "how good is the resulting state" is intuitive, and each sub-question admits a type of supervision that a monolithic Q-function cannot directly use.
- The evaluator is a drop-in replacement and is shown with two structurally different refinement methods (residual and latent steering).
- Real-robot experiments are included, and the negative result on Threading is reported and discussed honestly.
- The attention-readout visualization (Appendix B) is a useful qualitative look at what progress distillation changes in the value representation.
Critical weaknesses
W1. The claimed mechanism is not isolated, and the comparison is not controlled (main concern)
The paper's central claim is that decomposing the evaluator into dynamics and state value generalizes better to nearby unseen actions than a direct Q-function. Since V∘f and Q are equivalent in what they can represent, any advantage must come from either (a) the structure itself (e.g., dense next-state targets for f; V reusing states reached via other trajectories), or (b) access to pretrained components and extra supervision that the baseline does not receive. The experiments do not distinguish these.
(a) The comparison is not controlled. Sec. 4.1 states that the comparison "isolates the effect of replacing direct action-value estimation with dynamics–value decomposition." Tables 2–3 contradict this. Besides the evaluator, the two arms differ in:
- visual representation: IMPALA trained from scratch vs. frozen DINOv2 with a learned readout;
- critic initialization: random vs. RoboMeter progress for DICE-RL;
- batch size (256 vs. 64), update frequency, gradient steps per update, target update rate, ensemble size and aggregation (10 heads/min vs. 3 nets/mean), DrQ augmentation, and replay size (DSRL);
- training budget: Threading/DICE-RL uses 600k steps for Q vs. 300k for the proposed method, which also makes "final-checkpoint" comparisons ambiguous;
- task-specific tweaks applied only to the proposed method: online progress distillation for Square D0 only, and segment-level readout pretraining for Square D1 and Threading (DSRL).
(b) Supervision is unequal, all in favor of the proposed method. The proposed method receives RoboMeter progress labels, pretrained DINOv2 features, next-state prediction targets, and, in the real-world experiments, additional expert demonstrations for progress distillation. The DICE-RL direct-Q baseline receives only sparse TD targets on a randomly initialized critic. So it is not shown that, under equal supervision, the decomposed evaluator generalizes better.
(c) There is no direct evidence for the generalization claim. No experiment measures how accurately either evaluator scores nearby unseen actions, which is the core claim illustrated in Fig. 1. Appendix B compares two state-value targets (progress vs. MC returns), not decomposed vs. direct evaluators. If anything, it shows that denser labels help regardless of decomposition, which reinforces the confound.
(d) Plausible alternative explanations are not ruled out:
- pretrained frozen features vs. a critic encoder learned from sparse reward;
- value-guided best-of-16 selection with a progress-aware scorer from the very first episode, while the baseline ranks candidates with a randomly initialized Q-function;
- training stability: the V-TD target involves no maximization over actions, which may avoid Q-overestimation independent of any generalization argument.
I ask the authors to either (i) provide controlled comparisons with matched hyperparameters and budgets, including:
- a direct Q-function on the same frozen DINOv2 features and readout, regressed to RoboMeter progress of s_{t+H} (and optionally with a next-state auxiliary loss);
- the proposed method with randomly initialized V (no RoboMeter);
- the base policy with best-of-16 selection by S_DV and no RL updates;
- an offline evaluation of ranking/prediction accuracy of each evaluator on perturbed actions a + δ against held-out rollouts;
or (ii) reframe the contribution without the generalization claim, e.g., as a practical way to use pretrained world and progress models in policy refinement. Even under (ii), the hyperparameter and budget differences need to be removed.
W2. The generalization mechanism is underdeveloped
The argument for why the decomposition should generalize better appears only in passing. Sec. 3.2 notes that "every rollout transition provides a target for the dynamics model," and Sec. 3.3 that "both successful and failed rollouts provide dense supervision." But neither is connected to the generalization claim, which is stated as "local compositional generalization" without explanation. The density argument for V rests on RoboMeter labels rather than on the decomposition. Moreover, dense next-state supervision does not guarantee that f is sensitive to the action. With offline data from a single base policy, action variety per state is limited, and f could predict s' mostly from s while responding weakly or incorrectly to a. Policy refinement depends exactly on ∂f/∂a, yet no measure of action sensitivity or counterfactual prediction accuracy is reported.
W3. Technical soundness of the evaluator
- Dropping the reward/termination term (Eq. 4 → Eq. 3). The justification is that non-terminal chunks dominate. However, the chunk in which success occurs is precisely where the precise corrections targeted by the paper matter. On that chunk, V is evaluated on a predicted terminal state. TD never trains V on terminal states as inputs, so V there is anchored only by the initial progress distillation (online distillation is used only on Square D0) and may drift.
- What V estimates. V is trained by TD on replay data mixing offline base-policy rollouts and older online policies, with no action-dependent correction. It therefore estimates the value of the data-collecting policies, not the current or optimal policy, making the method closer to one-step policy improvement than to Q-learning. This should be discussed.
- Bounding the correction and model exploitation. Eq. 13 states that α keeps candidate actions within the region covered by the dynamics model, but Table 2 lists "residual scale 1, no norm cap" with only an L2 penalty. Nothing explicit prevents the actor from exploiting errors in f.
- Time-limit truncation is treated as terminal without bootstrapping (Tables 2–3). This applies to both arms, but it biases value targets.
W5. Novelty
Evaluating actions as V(f(s, a)) and backpropagating through a learned model is established (SVG(1)/value gradients, IVG, Dreamer, TD-MPC, model-based value expansion). RISE also combines a world model with a progress value model. The paper acknowledges this and positions its contribution as component-wise initialization for policy refinement. That places the burden on rigorous empirical evidence, which W1–W4 find lacking.
Questions for authors
- Which parts of f are actually pretrained? Sec. 3.3 says only the prediction head of a pretrained DINO-WM is fine-tuned, but Appendix A.2 describes substantial architectural changes (1-D conv action encoder, learnable query tokens, proprioception head), and Fig. 5 varies "DINO-WM training steps" from 50k to 300k. Is the predictor trained from scratch on D_off over frozen DINOv2 features?
- Does the DSRL direct-Q baseline also receive RoboMeter initialization? Table 3 lists "RoboMeter progress (offline)" without distinguishing the two arms.
3 comments
[W1(d)] Alternative explanations.
Our results argue against the first two explanations and only partly against the third. Frozen features do not help Q (within 3 points of IMPALA-based Q on DICE-RL Square D0 and D1). Selection without RL nearly matches the trained gain only on HammerCleanup (+12 versus +13 points), and there DICE-RL+DVD with a randomly initialized V, which starts without a progress-aware scorer, matches the full DVD. DSRL, which performs no selection, also gains from DVD.
On the third, Q does not overestimate: taking the minimum over its heads, it underestimates realized returns by 0.01 to 0.11. This pessimism may still slow Q independently of generalization, which our controls do not isolate.
[W2] Underdeveloped generalization mechanism.
Q learns an action's effect only from sparse TD targets on executed actions, whereas the dynamics model learns it from dense next-state targets on every transition, and the value model can score a predicted state that other trajectories have reached. The density argument does not rest on RoboMeter alone: with a randomly initialized V, DICE-RL+DVD and DSRL+DVD retain most of the gain.
[W3.1] Dropping the reward/termination term.
Dropping the term rarely changes the chosen chunk (DICE-RL, Square D0). On the chunk that completes the task, DVD scores the chosen chunk at 0.88 instead of 1, but in 94-97% of these decisions all 16 candidates succeed, so the choice does not change, and the value model does not drift on real terminal frames. One decision earlier, DVD skips a chunk that would complete the task immediately in 7% of successful episodes (1-5% for DICE-RL+Q), which delays success but does not prevent it.
[W3.2–W3.4] Value target, bounding the correction and time-limit truncation.
The value model indeed estimates the value of the data-collecting policies, so the method is closer to one-step policy improvement, and our comparison is between one-step policy improvement and off-policy Q-learning. In practice, against the evaluated policy's return, its bias stays within -0.03 to +0.02 in DSRL and -0.06 to 0.00 in DICE-RL.
The text at Equation (13) overstates how the correction is bounded. With residual scale 1 and no norm cap, only the L2 penalty and best-of-16 selection bound the correction. We have not directly tested whether the actor exploits errors in the dynamics model.
Treating time-limit truncation as terminal biases both critics' targets near the time limit in the same way, since both critics use this treatment.
[W5] Novelty: V(f(s,a)) and model-based value gradients are established.
We revised Section 2 (L124-L147) to state that DVD builds on the value-gradient mechanism of SVG and IVG. Our contribution is using it as a drop-in critic that improves existing online refinement methods for frozen pretrained policies.
IVG [1] is the closest work. It learns its policy, value and reward from scratch, mostly with shaped rewards, and its IVG(1) variant scores actions as V(f(s,a)) plus a learned reward. DVD scores actions like IVG(1) without the reward term, to refine a frozen pretrained policy online under sparse reward. RISE [2] keeps its world model and value model frozen and does not interact online, whereas DVD trains both online. Within 600k steps, DICE-RL+DVD outperforms DICE-RL+Q by 11.3 points on Square D0.
[Q1–Q2] Pretrained parts of f, and RoboMeter initialization for DSRL+Q.
In the simulated experiments, only the frozen DINOv2 encoder is pretrained. There, the predictor is trained from scratch on base-policy rollouts, then fine-tuned online on replay that includes demonstrations. "DINO-WM" names only the architecture family; apart from the frozen DINOv2 encoder, the dynamics model is not pretrained.
Neither Q baseline, including DSRL+Q, receives RoboMeter initialization: both start from random weights.
References
[1] Byravan, A., et al. (2020). Imagined value gradients: Model-based policy optimization with transferable latent dynamics models. CoRL 2020.
[2] Yang, J., et al. (2026). RISE: Self-improving robot policy with compositional world model. arXiv:2602.11075.
Dear Reviewer 1uKU,
Thank you again for your detailed feedback. If you have a chance to read our rebuttal, we’d appreciate hearing whether it addresses your concerns. We’d be happy to clarify any points that remain unclear.
Thank you for the detailed review and the concrete controls you proposed. Below, DVD is our critic V(f(s,a)), composed of a dynamics model (f) and a state-value model (V) initialized from RoboMeter task-progress labels, and Q is the original critic of DICE-RL or DSRL, which refine a pretrained base policy.
[W1] The claimed mechanism is not isolated.
We ran the four controls you requested and an equal-budget comparison in W1(a). The gain largely survives the controls below, and the ranking test on shifted actions favors DVD, as shown in W1(c).
Together, these controls support our contribution: DVD’s gain does not come from its pretrained components alone, but from replacing the direct Q critic with a decomposed evaluator trained online. Trained online, DVD improves DICE-RL and DSRL, and its critic ranks nearby unseen actions more accurately than Q (Section 4.3, Figure 5a).
[W1(a)] The comparison is not controlled.
We agree that the submission’s training budgets differed between variants. All main variants now train for 600k environment steps without online distillation, which had no significant effect, and we report the mean of the last three evaluations. This removes the budget mismatch and the final-checkpoint ambiguity. DVD outperforms Q by 11 to 18 points on six of the eight method-task pairs (two RL methods on four tasks; revised Figure 3). On Threading, DICE-RL+Q is 10.7 points better, and the DSRL variants tie.
When we rerun DSRL+Q on ManiSkill3 LiftPegUpright with DVD’s batch size, update cadence, critic count and replay size, it reaches 82.0% from ID and 11.5% from OOD initial states (80.4% and 10.3% with its default settings, Table 1), against 88.7% and 46.3% for DSRL+DVD. On MimicGen, hyperparameters remain only partly matched (Tables 3 and 4): DVD runs its dynamics model in every update, which limits its batch size to 64 on the same GPU (256 for Q), and the two also differ in update cadence, target update rate, critic ensemble, DrQ augmentation and, in DSRL, replay size and readout pretraining. Q keeps the published DICE-RL and DSRL recipes, while DVD’s settings follow from its cost and design: its dynamics model runs in every update, which also limits its update cadence, and its value model reads frozen DINOv2 features, so there is no critic encoder to augment.
[W1(b)] Supervision is unequal.
We agree. Our controls give Q the DINOv2 features and RoboMeter initialization of DVD, which does not help Q, and remove RoboMeter from DVD, which preserves most of its gain (W1). We have not run DVD without its DINOv2 features, and on the real robot its value model also receives expert demonstrations. We therefore view the real-robot comparison as a feasibility result under unequal supervision rather than a controlled comparison.
[W1(c)] Direct evidence for generalization.
We ran this test and added it to the revised paper (Section 4.3, Figure 5a).
At the last DSRL checkpoint (600k), the composed evaluator ranks nearby unseen actions correctly with an accuracy of 0.60, against 0.49 for Q, which is at chance. Its lead of 0.11 (95% CI 0.07 to 0.15) holds on all four tasks, significantly on three of them. From held-out states, we shift the executed 8-action chunk along each of the six end-effector axes in turn (x, y, z and three rotations) to nine candidates within ±0.4 and score each by its true return from 16 rollouts that use the same random seeds for every candidate.