acceptodds
Under review as a conference paper at ICLR 2027

Uncertainty-Guided λ-Soft Value Estimation for Sample-Efficient Maximum Entropy Reinforcement Learning

Abstract

Maximum entropy actor-critic methods such as Soft Actor-Critic (SAC) are known for effective exploration and robustness, yet their learning progress hinges on a learned soft critic whose estimates are inaccurate early in training. Model-based reinforcement learning (MBRL) promises higher sample efficiency, but compounding model errors limit how far learned dynamics can be trusted. In this work, we use model-based rollouts to improve the sample efficiency of SAC. We introduce -soft value estimation, which recursively blends the critic's direct estimate with a model-based bootstrapped estimate at each step of a model rollout, and prove that under an exact model such estimates accelerate soft policy iteration, with the one-step backup of SAC as the slowest case. To set the per-step blending factor automatically, we propose uncertainty guidance, which balances critic-ensemble disagreement against dynamics-ensemble disagreement: the estimate relies on the model where the critic is uncertain and falls back to the critic for the rest of the rollout once only the model is uncertain. On four contact-rich MuJoCo tasks, -SAC is competitive in sample efficiency, wall-clock time, and robustness to tuning, as a single hyperparameter setting solves all tasks, and it reaches a final return on par with model-free methods.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Model-based Reinforcement Learning, Actor-Critic, Entropy Regularisation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tfMFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes λ-SAC, a model-based maximum-entropy actor–critic algorithm. At each model-rollout node, a soft λ-return combines predicted rewards with critic estimates. The mixing weight depends on disagreement among dynamics models and among critics. The resulting return estimate is used both to train the critic and to update the actor through differentiable model rollouts. The paper proves fixed-point and contraction properties under an exact model and a fixed policy, and evaluates the method on four Gymnasium MuJoCo tasks.

Critical strengths
  1. The paper addresses a central difficulty in model-based actor–critic learning: longer model rollouts reduce reliance on an inaccurate critic but accumulate dynamics errors. Its distinctive contribution is to combine model and critic disagreement in a soft λ-return, then use that same estimate for both critic targets and actor updates through the model. This provides a coherent way to adapt reliance on imagined experience while retaining the maximum-entropy objective. The fixed-point and contraction analysis further explains why the estimator preserves the desired soft value and can accelerate policy evaluation under an exact model. The combination of adaptive blending, shared use in evaluation and improvement, and an explicit theoretical foundation is a meaningful methodological contribution.
  2. The complete system shows promising practical benefits rather than only an improvement to an isolated estimator. With one hyperparameter configuration, it learns across four MuJoCo tasks and shows strong sample efficiency on Walker2d and Humanoid relative to the tested model-free baselines. The main performance curves report five-seed statistics, and the evaluation considers both environment interactions and training time. These results support the usefulness of the overall approach within the evaluated setting.
Critical weaknesses
  1. The ablations do not isolate the benefit of state–action-dependent guidance. Guided λ varies both across state–action pairs and over training, whereas fixed λ does neither. Figure 7 shows that the guided effective horizon starts shorter and increases over training. Its advantage could therefore come from a simple short-to-long rollout schedule, without requiring different cutoffs for different state–action pairs. Matching this training trend with a state-independent schedule would test the paper's claim that the cutoff location matters.
  2. The experiments provide limited evidence of reliable gains and broad applicability. (1) Key component ablations use only one seed, including the matched four-critic SAC/neither control, while the main SAC baseline uses two critics. (2) Several MBPO curves stop before the stated 400k-step budget. (3) The comparison lacks recent representative methods from the same research setting. (4) Only four Gymnasium MuJoCo tasks are tested. Together, these gaps limit confidence in the stability of the gains, their relevance against contemporary methods, and the transfer of the shared hyperparameter setting across tasks.
  3. (Minor) Taking the minimum over all critics ties the degree of pessimism to ensemble size and can lead to excessive underestimation as more critics are added. Consequently, the ensemble-size ablation changes both uncertainty estimation and value aggregation, making its results difficult to interpret. A mean-minus-uncertainty bound [1] would allow separate control of pessimism. The gate also does not explicitly account for aleatoric uncertainty in stochastic-policy returns [1].

[1] Yu et al. Taming Aleatoric Impulse in Off-Policy Reinforcement Learning. ICML, 2026.

Questions for authors
  1. Can the authors compare guided λ with a state-action-independent schedule matched to its effective horizon over training? These controls would clarify whether the gains require state-action-dependent cutoffs.
  2. Can the authors provide multi-seed results for every curve in all three panels of Figure 3, extending all currently single-seed variants and reporting the seed counts and variability? Please also provide complete MBPO curves, a recent representative baseline, and results on a small number of additional tasks under comparable budgets. A focused extension would be sufficient.
  3. The theory establishes contraction under an exact model, but does not show whether the estimator reduces value-estimation error with learned dynamics. The reported returns and training losses also do not directly establish lower value-estimation bias. Can the authors provide either further theoretical analysis explaining when the estimator improves accuracy, or a focused empirical comparison of its bias against that of the critic alone? For example, Monte Carlo estimates of discounted soft returns from the same state–action pairs under a frozen policy could provide a reference for measuring overestimation and underestimation on an existing task.
2 comments

We thank the reviewer for the detailed feedback, which we used to extend the ablations, extend the contraction rate proof, and sharpen the positioning of the paper.

Q1: state-action-independent schedule

  • Added: a schedule matched to the guided effective λ over training, 4 tasks × 5 seeds (Fig. 2, col. 4).
  • It matches the guidance on three of four tasks: the gain comes from the task-specific level of λ, not from state-action-dependent cutoffs. We removed the claim that the cutoff location matters and re-weighted the contributions (see response to ASe8).

Q2: multi-seed results, MBPO, baselines

  • All Fig. 3 variants, including the four-critic SAC control, now run with 5 seeds on all 4 tasks (former Fig. 3, now Fig. 2; table in our response to ASe8).
  • MBPO: we reran the benchmarks with more compute; the revised Fig. 1 shows the extended curves.
  • The closest methods (STEVE, MAAC, SHAC, Dreamer) are compared by their estimators in our response to ASe8.

Q3: accuracy under learned dynamics

  • We added an analysis for learned dynamics to the appendix (proof at the end of this reply). It complements Props. 1–2 and leaves the reported results unchanged.
  • Result: for any λt\lambda_t, the λ-soft estimate is no worse than the critic alone if, at every node, the dynamics error seen through the value is small relative to the critic error: γεt≤(1−γ)δ\gamma\varepsilon_t \le (1-\gamma)\delta.
  • Implications:
    • Early in training, a poor critic (large δ\delta) relaxes the condition, so even a coarse model helps; as the critic improves, the model has to improve too. This is consistent with the guided λ rising over training, as the reviewer noted.
    • The required model accuracy depends on the sharpness of the value landscape, which is unknown a priori. This motivates setting λ from data instead of tuning it per task (cf. lambda guidance criticms).
    • The condition is in value units, whereas the disagreement statistics are uncalibrated proxies; per-node cutoffs therefore need a calibrated error statistic (Sec. 7).

Minor: pessimism and ensemble size

  • We agree that the minimum over NN critics couples pessimism to NN; we note this for the ensemble-size ablation and cite the mean-minus-uncertainty bound of Yu et al. (2026), suggested by the reviewer, as a way to separate the two.

Proof (Q3)

Definitions

  • Vπ(s)=Ea∼π[Qsoftπ(s,a)−αlog⁡π(a∣s)]V^\pi(s) = \mathbb E_{a\sim\pi}[Q^\pi_{\text{soft}}(s,a) - \alpha\log\pi(a|s)]: true soft value of π\pi under the true dynamics; used in the analysis only
  • δ=sup⁡s,a∣min⁡iQθi(s,a)−Qsoftπ(s,a)∣\delta = \sup_{s,a} |\min_i Q_{\theta_i}(s,a) - Q^\pi_{\text{soft}}(s,a)|: worst-case error of the critic bootstrap
  • εt=∣Vπ(f^(xt,at))−Vπ(f(xt,at))∣\varepsilon_t = |V^\pi(\hat f(x_t, a_t)) - V^\pi(f(x_t, a_t))|: dynamics error at node tt, measured as the value difference between the predicted and the true next state
  • et=sup⁡∣Qtλ−Qsoftπ∣e_t = \sup |Q^\lambda_t - Q^\pi_{\text{soft}}|: error of the blended estimate at node tt

Sketch

  • Write out the difference, using Qπ=(1−λt)Qπ+λt(rt+γVπ(xt+1))Q^\pi = (1-\lambda_t)Q^\pi + \lambda_t(r_t + \gamma V^\pi(x_{t+1})), and add and subtract Vπ(x^t+1)V^\pi(\hat x_{t+1}) in the rollout part:

Qtλ−Qπ=(1−λt)(Qθ,t−Qπ)+λtγ(Vλ(x^t+1)−Vπ(x^t+1))+λtγ(Vπ(x^t+1)−Vπ(xt+1))Q^\lambda_t - Q^\pi = (1-\lambda_t)(Q_{\theta,t} - Q^\pi) + \lambda_t\gamma(V^\lambda(\hat x_{t+1}) - V^\pi(\hat x_{t+1})) + \lambda_t\gamma(V^\pi(\hat x_{t+1}) - V^\pi(x_{t+1}))

  • The three terms are bounded by δ\delta, et+1e_{t+1}, and εt\varepsilon_t; for the second, Vλ−Vπ=Ea[Qλ−Qπ]V^\lambda - V^\pi = \mathbb E_a[Q^\lambda - Q^\pi], the entropy terms cancel. Taking the sup over the left side:

et≤(1−λt)δ+λtγ(et+1+εt)e_t \le (1-\lambda_t)\delta + \lambda_t\gamma(e_{t+1} + \varepsilon_t)

  • Backward over the nodes: inserting et+1≤δe_{t+1} \le \delta gives et≤(1−λt)δ+λtγ(δ+εt)e_t \le (1-\lambda_t)\delta + \lambda_t\gamma(\delta + \varepsilon_t), which is ≤δ\le \delta iff γεt≤(1−γ)δ\gamma\varepsilon_t \le (1-\gamma)\delta.
  • Hence, if γεt≤(1−γ)δ\gamma\varepsilon_t \le (1-\gamma)\delta at every node, then e0≤δe_0 \le \delta for any λt\lambda_t.

I thank the authors for the additional experiments and theoretical analysis, which address most of my concerns. However, the experimental scope remains limited, particularly in terms of task diversity and comparisons with recent methods. I therefore maintain my original rating.

Reviewer_j5zhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes λ-SAC, which blends soft critic estimates with model-based rollouts for both critic training and actor updates. The blending weights depend on disagreement within dynamics and critic ensembles. Under an exact model, the authors show that the evaluation operator preserves the soft value fixed point and improves its contraction rate. Experiments on four MuJoCo tasks report improved sample efficiency.

Critical strengths

Using the same λ-soft estimator for critic targets and actor gradients gives a coherent method. The fixed-point and contraction analysis provides useful justification for policy evaluation under an exact model. The main experiments use five seeds and one hyperparameter setting across four tasks. The reported gains on Walker2d and Humanoid suggest practical value for improving SAC’s sample efficiency.

Critical weaknesses

The evidence for joint uncertainty guidance is limited. All ablations use only one seed on Humanoid, and the confidence-scale study does not include the model-only gate, κQ = 0. Therefore, it is unclear whether using both uncertainties consistently improves over simpler alternatives. The comparisons do not fully establish the method’s advantage. MAAC, which also differentiates through model rollouts, is discussed but not evaluated. The wall-clock comparison uses one seed and mixes different UTD ratios and implementation optimizations, including CUDA graphs only for λ-SAC. These factors make the source of the efficiency gains difficult to assess.

Questions for authors
  1. Can you provide results across multiple seeds for joint guidance versus model-only gating and fixed λ? Does the advantage also hold on another task?

  2. How much of the wall-clock advantage remains with comparable implementation optimizations and controlled update budgets? A breakdown of dynamics training, critic updates, and actor updates would help.

Reviewer_ASe8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the bias trade-off between model error and critic error in model-based reinforcement learning. The authors propose λ\lambda-soft SAC, which dynamically adjusts the blending parameter λ\lambda based on uncertainty estimates of both the dynamics model and the critic. The proposed λ\lambda-soft value estimator is incorporated into policy evaluation and policy improvements. The performance is evaluated on four MuJoCo tasks to demonstrate competitive sample efficiency.

Critical strengths

The proposed method extends λ\lambda-blending to policy evaluation and policy improvement in SAC using a λ\lambda-soft value estimator. The blending parameter lambda is adaptively adjusted according to the uncertainty estimates from both dynamics and critic ensembles, balancing the reliance on model rollouts and critic bootstrapping. The author also provides theoretical results showing that the proposed method has a faster contraction rate compared to standard SAC under an exact dynamics model.

Critical weaknesses

Limited methodological novelty and insufficient ablation study

The proposed method mainly combines several existing techniques, including λ\lambda-blending (Bhardwaj et al., 2021), uncertainty-guided value estimation (STEVE, 2019), and differentiable model-based policy improvement (MAAC and Dreamer). The main novelty appears to be the integration of these techniques into SAC, particularly the joint use of dynamics and critic uncertainties to adjust blending parameter.

Due to limited methodological novelty, I would expect more detailed ablation studies on the effect of each component. Although Figure 3 shows the benefits of using λ\lambda-soft estimation to both policy evaluation and policy improvement, and using uncertainty guidance over fixed or random λ\lambda, it does not isolate the contributions of dynamics and critic uncertainties. In particular, a comparison between joint uncertainty guidance, dynamics-only guidance, critic-only guidance is missing. Therefore, it remains unclear whether jointly considering both uncertainties provides additional benefits.

Questions for authors
  1. Why do MBPO learning curves in Figure 1 terminate before reaching the 400k environment-step budget? It is a bit weird to see curves get cut off while still increasing.

  2. Ablation comparisons in Figure 3 only have one seed each, making it difficult to assess the statistical significance and robustness of the improvements.

2 comments

We thank the reviewer for the detailed feedback, which we used to extend the ablations and sharpen the positioning of the paper.

Summary

  • As the reviewer notes, the method extends λ-blending to both policy evaluation (PE) and policy improvement (PI). This goes beyond integrating existing techniques: we extend SAC's soft policy iteration to λ-soft policy iteration, where one fully soft (max-ent) multi-step estimator is both the critic target and, differentiated through the learned dynamics, the policy objective.
  • SAC uses a one-step soft estimate without a model; MAAC and SHAC use two different, not fully soft estimators in the two steps; STEVE uses its estimate only in the critic target.
  • The new ablations (4 tasks × 5 seeds) show that this unification drives the gains.

Effect of each component (former Fig. 3, now Fig. 2, 5 seeds on 4 tasks)

Mean return over 0–400k (5 seeds)HopperWalker2dHumanoidPusher
λ-SAC (full)2782 ± 2154565 ± 4213558 ± 499−34.6 ± 1.3
estimator in PE only1827 ± 8023378 ± 8022820 ± 418−37.1 ± 2.2
estimator in PI only1752 ± 9152323 ± 8771788 ± 207−41.1 ± 3.6
neither (SAC, 4 critics)2061 ± 7654104 ± 5241247 ± 195−43.8 ± 1.6
no gradient through model2117 ± 10404353 ± 7672087 ± 484−36.1 ± 1.7
dynamics-only guidance3042 ± 2554767 ± 8113487 ± 268−33.2 ± 0.6
critic-only guidance1532 ± 6963977 ± 2971922 ± 188−43.9 ± 2.2
fixed λ, matched level2738 ± 2574746 ± 6123419 ± 420−32.4 ± 1.1
fixed λ = 0.51841 ± 10024112 ± 4892829 ± 668−34.7 ± 0.9
  • The estimator in both PE and PI is best on all tasks; in one step only, it can fall below SAC (Walker2d).
  • The gradient through the model contributes most on Humanoid and Hopper.
  • Requested guidance ablation: dynamics-only matches joint guidance; critic-only is slowest. A fixed λ at the matched level matches the guidance, while λ = 0.5 is slower: the level of λ matters, and the dynamics ensemble sets it per task (0.83 to 0.96) under one hyperparameter setting. With the current disagreement statistic, the critic term adds no measurable benefit.
  • Why λ is set from data: our analysis for learned dynamics (response to tfMF, Q3) shows that the model accuracy at which the rollout helps depends on the sharpness of the value landscape, which is unknown a priori. The guidance therefore finds the task-specific λ within a single run, instead of a per-task search.

Novelty

  • One fully soft (max-ent) multi-step estimator, with the entropy term at every rollout node, used consistently in both steps: as the critic target in policy evaluation, and as the policy objective in policy improvement, differentiated through the learned dynamics model. Under an exact model, it keeps SAC's soft fixed point and contracts faster than SAC's backup (Prop. 1–2).
  • Prior work uses a model-based estimate in one step only, two different estimators, no entropy, or no learned actor (Bhardwaj et al., MPC):
Max-ent targetPE (critic target)PI (actor objective)Same estimator in PE and PI
STEVE✗ensemble-weighted HH-step targetscritic only (DDPG)✗
MAACentropy bonus onlySTEVE-style targetsHH-step return through model, terminal QQ✗
Dreamer✗λ\lambda-return on V(s)V(s), imagination onlysame λ\lambda-return✓
SHAC✗TD(λ\lambda) on V(s)V(s)hh-step return through simulator, terminal VV✗
λ-SAC (ours)✓ at every nodeλ\lambda-soft estimate on Q(s,a)Q(s,a)same λ\lambda-soft estimate, through model✓

Revisions

  • Re-weighted contributions: λ-soft policy iteration as the main contribution (abstract, introduction); uncertainty guidance as task-specific λ selection, not per-node cutoffs (contributions, Sec. 6.3, conclusion); a calibrated per-node error statistic named as a limitation (Sec. 7).
  • MBPO (mbrl-lib, UTD 20) needs more than 48 h per run for 400k steps; the original runs ended at our compute time limit. We rerun them to 400k steps for the final version.

Thank you for the additional ablations. However, my novelty concern remains. The estimator builds on established λ-return and soft value-expansion methods, and the experiments do not clearly isolate the benefit of the proposed unified formulation. The theoretical guarantees are also limited to fixed-policy evaluation under exact dynamics, without addressing the coupled actor–critic learning process. I therefore still find the contribution incremental and maintain my weak-reject recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.