acceptodds
Withdrawn from ICLR 2027

DRIVE: Decoupled Representation with Informed Value Estimation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Model-based reinforcement learning (MBRL) promises sample-efficient continuous control, yet existing methods face a fundamental tension between representation quality and training stability. Reconstruction-based approaches allocate encoder capacity to value-irrelevant features, while joint dynamics-value supervision lets noisy bootstrapped gradients distort the learned latent geometry. We introduce DRIVE (Decoupled Representation with Informed Value Estimation), a framework built on one principle: representation learning should be decoupled from value-driven optimization. DRIVE isolates the encoder from all critic and actor gradients, training it exclusively through structured supervision, including dynamics prediction, contrastive temporal consistency, reward estimation, and cumulative return prediction on the unit hypersphere. The critic fits values on the resulting well-conditioned latent space, and the actor combines single-step Q-gradients with a multi-step soft-decay rollout through the learned dynamics, internalizing trajectory-level planning at training time with no additional inference cost. Across 28 high-dimensional continuous control tasks, DRIVE matches or exceeds state-of-the-art model-based and model-free baselines. Ablation studies confirm that the soft-decay rollout and structured encoder supervision provide complementary gains. Crucially, our analysis reveals a fundamental failure mode in high-dimensional MBRL: local representation aliasing inevitably leads to Q-value flattening and actor gradient starvation, a pathology that DRIVE effectively cures.

Primary area:
reinforcement learning
Keywords:
model-based reinforcement learning, representation learning, contrastive learning, encoder design

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8xi6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents DRIVE, a state-based model-based reinforcement learning framework that decouples representation learning from actor and critic optimization. The encoder is trained with dynamics prediction, InfoNCE temporal contrastive learning, reward prediction, termination prediction, and cumulative return prediction, while its state representations are projected onto a unit hypersphere. The critic performs value estimation on detached latent representations, and the actor combines a single-step Q-gradient with a multi-step soft-decay rollout through the learned dynamics model. The method is evaluated on 28 tasks from DMC Hard, MyoSuite, and HumanoidBench, with additional ablations and representation diagnostics examining collapse, local aliasing, and actor gradient starvation.

Critical strengths
  1. The method is evaluated across a broad set of high-dimensional control tasks. DRIVE is evaluated on 28 tasks spanning DMC Hard, MyoSuite, and HumanoidBench. The reported results show strong performance on many challenging locomotion and musculoskeletal-control tasks, including substantial gains on DMC Humanoid and Dog tasks. The paper also reports learning curves, multiple random seeds, and representation-level diagnostics.

  2. The separation between representation learning, value fitting, and policy optimization is a coherent design direction. The proposed architecture assigns different roles to the encoder, critic, and actor, and it avoids inference-time MPPI or other online search procedures. If implemented as intended, this design could provide planning-related training signals while retaining a low-cost reactive policy at deployment.

Critical weaknesses
  1. The scalar reward head is not incorporated consistently into the stated training procedure. The paper introduces a scalar reward head rscr_{\mathrm{sc}}, states that it is trained with a Smooth-L1 loss using λrsc=0.1\lambda_{r_{\mathrm{sc}}}=0.1, and uses it in the soft-decay rollout. However, the total encoder objective in Eq. (8) does not contain a corresponding scalar-reward loss, and the hyperparameter table does not list a separate weight for this loss. In the encoder-training pseudocode, the scalar reward output is returned but assigned to _, so no corresponding optimization term is shown. The rollout pseudocode itself correctly selects the fifth scalar-reward output as pred_r, but the manuscript does not clearly specify how this head is trained. Since the rollout is a central component of the method, this missing training specification affects the interpretation of the rollout ablation.

  2. The stated role of the terminal Q-value is inconsistent with the gradient computation. Equation (12) includes a stop-gradient terminal value, sg⁡(Qψ1(z^t+Hsa))\operatorname{sg}(Q_{\psi_1}(\hat z_{t+H}^{sa})), and the reference implementation applies .detach() to the terminal Q-value. Under the stated objective, this term contributes no gradient to the actor and therefore acts only as an additive constant during actor optimization. This is difficult to reconcile with the claim that the terminal Q-value provides trajectory-level guidance beyond the rollout horizon. The authors should clarify whether they intend to freeze only the critic parameters while preserving gradients with respect to the actor input and action, or whether the terminal Q-value is intended only as a detached numerical estimate. The limiting-case interpretation of the soft-decay coefficient is also unclear. The paper states that α=0\alpha=0 recovers model-free actor-critic, while Appendix E states that the direct gradient from the predicted reward through the local action is not attenuated by soft-decay. Therefore, even when α=0\alpha=0, the rollout still contains model-based reward-gradient paths and is not strictly equivalent to a model-free actor update.

  3. The methodological novelty is not sufficiently separated from existing decoupled representation approaches. The paper combines gradient-isolated representation learning, temporal contrastive learning, auxiliary reward prediction, spherical normalization, and differentiable rollout guidance. Several of these components have substantial precedent, including the decoupled encoder and temporal contrastive objectives discussed in the paper’s comparison with TCRL and MR.Q. The soft-decay operation itself is a simple stop-gradient interpolation that scales gradient flow. The paper does not compare it against alternatives such as truncated backpropagation, adaptive gradient attenuation, or other methods for stabilizing multi-step model-based actor training. Consequently, the central conceptual contribution is difficult to distinguish from a combination of existing techniques.

  4. The theoretical and experimental evidence does not fully support the strength of the paper’s claims. Proposition B.3 establishes that complete global collapse gives the InfoNCE baseline loss, but it does not guarantee that local representation aliasing is avoided. Similarly, the Lipschitz Bellman-error bound in Appendix F is a standard conditional bound and does not by itself show that gradient isolation reduces the relevant Lipschitz constants or improves value estimation. The causal chain from local aliasing to Q-value flattening and actor gradient starvation is mainly demonstrated on the Dog Run ablation. The component ablations are also limited to Dog Run, H1-Run, and Myo-Key-Turn-Hard. The soft-decay coefficient α=0.5\alpha=0.5, rollout horizon, rollout weight, and encoder-loss weights are fixed without sensitivity analysis. In addition, the baseline results are collected from different sources and use different seed counts, which makes it difficult to isolate the contribution of DRIVE. The paper also reports a DMC suite-level normalized score of 1.13 for DRIVE in the text, whereas Figure 2 displays a value of 1.00 under the stated min-max normalization; this discrepancy should be clarified.

Questions for authors
  1. Is a scalar-reward loss LrscL_{r_{\mathrm{sc}}} included in the actual encoder update? Please provide the complete objective and clarify how the scalar reward head is optimized.

  2. If the terminal Q-value is detached as shown in Eq. (12) and in the reference implementation, how can it influence the actor update? Is the intended operation to freeze the critic parameters while retaining gradients with respect to the actor input?

  3. How sensitive are the results to α\alpha, the rollout horizon, the rollout loss weight, and the encoder-loss weights? Can the authors compare the soft-decay mechanism with a matched truncated-BPTT or adaptive-gradient baseline?

  4. Can the authors clarify the normalization protocol used for the suite-level scores, particularly the difference between the DMC value reported in the text and the value shown in Figure 2?

Reviewer_UdswOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on model based reinforcement learning (MBRL). It mainly addresses the representation learning part and proposes to decouple the state encoder training from the value function training so that their gradients would not conflict. Also, the authors add contrastive loss in the representation learning. Lastly, they propose a soft-decay rollout for combining both the gradient of the rollouts and rollouts without gradients for planning. Experiments on 28 continuous control tasks from DeepMind Control Suite, MyoSuite, and HumanoidBench report strong performance compared with TD-MPC2 and several model-free baselines.

Critical strengths
  1. The paper presents a well-motivated representation learning framework. The paper addresses an important challenge in model-based reinforcement learning: learning representations that are simultaneously suitable for dynamics prediction and value estimation. Separating encoder optimization from actor-critic learning is a reasonable approach for reducing interference between different training objectives.

  2. The soft-decay rollout provides an intuitive compromise between full backpropagation through learned dynamics and purely value-based policy optimization. The actor can exploit imagined trajectories during training while maintaining a simple feedforward policy during inference.

  3. The paper is clear and writing is easy to follow.

Critical weaknesses
  1. The main contribution claimed in the paper is the decoupling between the encoder learning and the value function learning. However, the encoder is still trained together with predicting rewards. Does not this training break the claim?

  2. The training combines many losses. It could be hard to select the weights for each loss. In this way, the method may not be easy to use.

  3. The experimental results are not so exciting. Looking at Figure J.1: Learning curves on H1Bench tasks, the performance of the proposed DRIVE varies a lot and it works badly on several tasks.

Questions for authors
  1. Since the core claim is about learning good representation for high-dimensional control tasks, it would be more convincing to test your method on image-based benchmarks, where the representation learning is more critical.

  2. Could you ablate the proposed soft-decay mechanism more? For example, how does it work with different gradient attenuation weights?

Reviewer_fVpfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces DRIVE, a model-based RL method for high-dimensional continuous control. The key idea is to decouple representation learning from value learning: the encoder is trained only via direct supervised losses (dynamics, contrastive, reward, and cumulative-return prediction), using stop-gradients for the critic and actor. The critic therefore fits values in a latent space unaffected by bootstrapped Bellman noise. Value information reaches the encoder mainly through the cumulative-return prediction head. The actor is trained with single-step Q-gradients plus a soft-decay rollout through the learned dynamics, which attenuates gradients as αh\alpha^h over the horizon and adds no inference cost.

The paper identifies a failure mode in high-dimensional MBRL (local representation aliasing that flattens Q-values and starves actor gradients) and evaluates DRIVE on 28 tasks across DMC Hard, MyoSuite, and HumanoidBench against several baselines.

Critical strengths

Interesting idea. The paper identifies a tension in high-dimensional MBRL: reconstruction-based encoders that can waste capacity modeling value-irrelevant features, and joint dynamics-value supervision that can let noisy bootstrapped gradient distort the latent geometry. The paper also states a clear thesis: representation learning should be decoupled from value-driven optimization.

Design of the ablation study. The paper includes ablations that isolate the contribution of each building block by removing single components. The authors distinguish between "universally critical" and "task-dependent" components.

Critical weaknesses

Clarity of presentation. Several parts of the paper are more prose-heavy than the content requires, which makes it harder to follow than necessary. A more concise, direct presentation would significantly improve readability and clarity.

Evaluation protocol. For comparing DRIVE, the paper refers to baseline results from third-party sources under different protocols, seeds, and codebases. This makes the head-to-head comparisons, and especially the claim of beating SOTA approaches, unreliable without rerunning baselines under a consistent protocol. Independent of this, the hyperparameters of DRIVE are fixed for all tasks.

Limited statistical rigor. The paper reports results for DRIVE over only three seeds, against baselines with three to ten seeds from different sources, with no significance testing or confidence intervals on a high-variance domain.

The central claim is somewhat overstated. The paper frames local representation aliasing as a fundamental, inevitable failure mode of high-dimensional MBRL and claims DRIVE overcomes it, but the empirical evidence comes mainly from one task's diagnostic (Dog Run). So, it seems to be a plausible hypothesis, not yet a demonstrated general finding.

Reproducibilty. Although the paper reports the hyperparameters used for DRIVE and presents a reference implementation of DRIVE as pseudocode in the appendix, the actual source code (especially for the evaluation) and experimental data are not provided, which limits reproducibility.

Minor points that did not affect my overall recommendation:

  • Some figures appear to have relatively low resolution and may not be vector graphics.
  • Inconsistent notation in Figure 1.
  • There are line breaks within inline equations, for example in Lines 196 or 1088.

Currently, these concerns clearly outweigh the strengths of the paper.

Questions for authors
  • The paper attributes the w/o-InfoNCE collapse to local representation aliasing. What distinguishes "aliasing" from simply under-regularized or under-determined representations?
  • Since α\alpha places DRIVE between model-free actor-critic and full BPTT, how was α=0.5\alpha = 0.5 chosen, how sensitive are the results to it, and could an adaptive or learned α\alpha improve on the fixed choice?
  • The encoder's loss weights are fixed across all tasks. How were these chosen, and how sensitive are the results to them?
  • Given the high variance of the domain, why were experiments run over only three seeds? Can the authors provide significance testing or confidence intervals over more seeds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.