- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents DRIVE, a state-based model-based reinforcement learning framework that decouples representation learning from actor and critic optimization. The encoder is trained with dynamics prediction, InfoNCE temporal contrastive learning, reward prediction, termination prediction, and cumulative return prediction, while its state representations are projected onto a unit hypersphere. The critic performs value estimation on detached latent representations, and the actor combines a single-step Q-gradient with a multi-step soft-decay rollout through the learned dynamics model. The method is evaluated on 28 tasks from DMC Hard, MyoSuite, and HumanoidBench, with additional ablations and representation diagnostics examining collapse, local aliasing, and actor gradient starvation.
Critical strengths
-
The method is evaluated across a broad set of high-dimensional control tasks. DRIVE is evaluated on 28 tasks spanning DMC Hard, MyoSuite, and HumanoidBench. The reported results show strong performance on many challenging locomotion and musculoskeletal-control tasks, including substantial gains on DMC Humanoid and Dog tasks. The paper also reports learning curves, multiple random seeds, and representation-level diagnostics.
-
The separation between representation learning, value fitting, and policy optimization is a coherent design direction. The proposed architecture assigns different roles to the encoder, critic, and actor, and it avoids inference-time MPPI or other online search procedures. If implemented as intended, this design could provide planning-related training signals while retaining a low-cost reactive policy at deployment.
Critical weaknesses
-
The scalar reward head is not incorporated consistently into the stated training procedure. The paper introduces a scalar reward head , states that it is trained with a Smooth-L1 loss using , and uses it in the soft-decay rollout. However, the total encoder objective in Eq. (8) does not contain a corresponding scalar-reward loss, and the hyperparameter table does not list a separate weight for this loss. In the encoder-training pseudocode, the scalar reward output is returned but assigned to _, so no corresponding optimization term is shown. The rollout pseudocode itself correctly selects the fifth scalar-reward output as pred_r, but the manuscript does not clearly specify how this head is trained. Since the rollout is a central component of the method, this missing training specification affects the interpretation of the rollout ablation.
-
The stated role of the terminal Q-value is inconsistent with the gradient computation. Equation (12) includes a stop-gradient terminal value, , and the reference implementation applies .detach() to the terminal Q-value. Under the stated objective, this term contributes no gradient to the actor and therefore acts only as an additive constant during actor optimization. This is difficult to reconcile with the claim that the terminal Q-value provides trajectory-level guidance beyond the rollout horizon. The authors should clarify whether they intend to freeze only the critic parameters while preserving gradients with respect to the actor input and action, or whether the terminal Q-value is intended only as a detached numerical estimate. The limiting-case interpretation of the soft-decay coefficient is also unclear. The paper states that recovers model-free actor-critic, while Appendix E states that the direct gradient from the predicted reward through the local action is not attenuated by soft-decay. Therefore, even when , the rollout still contains model-based reward-gradient paths and is not strictly equivalent to a model-free actor update.
-
The methodological novelty is not sufficiently separated from existing decoupled representation approaches. The paper combines gradient-isolated representation learning, temporal contrastive learning, auxiliary reward prediction, spherical normalization, and differentiable rollout guidance. Several of these components have substantial precedent, including the decoupled encoder and temporal contrastive objectives discussed in the paper’s comparison with TCRL and MR.Q. The soft-decay operation itself is a simple stop-gradient interpolation that scales gradient flow. The paper does not compare it against alternatives such as truncated backpropagation, adaptive gradient attenuation, or other methods for stabilizing multi-step model-based actor training. Consequently, the central conceptual contribution is difficult to distinguish from a combination of existing techniques.
-
The theoretical and experimental evidence does not fully support the strength of the paper’s claims. Proposition B.3 establishes that complete global collapse gives the InfoNCE baseline loss, but it does not guarantee that local representation aliasing is avoided. Similarly, the Lipschitz Bellman-error bound in Appendix F is a standard conditional bound and does not by itself show that gradient isolation reduces the relevant Lipschitz constants or improves value estimation. The causal chain from local aliasing to Q-value flattening and actor gradient starvation is mainly demonstrated on the Dog Run ablation. The component ablations are also limited to Dog Run, H1-Run, and Myo-Key-Turn-Hard. The soft-decay coefficient , rollout horizon, rollout weight, and encoder-loss weights are fixed without sensitivity analysis. In addition, the baseline results are collected from different sources and use different seed counts, which makes it difficult to isolate the contribution of DRIVE. The paper also reports a DMC suite-level normalized score of 1.13 for DRIVE in the text, whereas Figure 2 displays a value of 1.00 under the stated min-max normalization; this discrepancy should be clarified.
Questions for authors
-
Is a scalar-reward loss included in the actual encoder update? Please provide the complete objective and clarify how the scalar reward head is optimized.
-
If the terminal Q-value is detached as shown in Eq. (12) and in the reference implementation, how can it influence the actor update? Is the intended operation to freeze the critic parameters while retaining gradients with respect to the actor input?
-
How sensitive are the results to , the rollout horizon, the rollout loss weight, and the encoder-loss weights? Can the authors compare the soft-decay mechanism with a matched truncated-BPTT or adaptive-gradient baseline?
-
Can the authors clarify the normalization protocol used for the suite-level scores, particularly the difference between the DMC value reported in the text and the value shown in Figure 2?