- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes λ-SAC, a model-based maximum-entropy actor–critic algorithm. At each model-rollout node, a soft λ-return combines predicted rewards with critic estimates. The mixing weight depends on disagreement among dynamics models and among critics. The resulting return estimate is used both to train the critic and to update the actor through differentiable model rollouts. The paper proves fixed-point and contraction properties under an exact model and a fixed policy, and evaluates the method on four Gymnasium MuJoCo tasks.
Critical strengths
- The paper addresses a central difficulty in model-based actor–critic learning: longer model rollouts reduce reliance on an inaccurate critic but accumulate dynamics errors. Its distinctive contribution is to combine model and critic disagreement in a soft λ-return, then use that same estimate for both critic targets and actor updates through the model. This provides a coherent way to adapt reliance on imagined experience while retaining the maximum-entropy objective. The fixed-point and contraction analysis further explains why the estimator preserves the desired soft value and can accelerate policy evaluation under an exact model. The combination of adaptive blending, shared use in evaluation and improvement, and an explicit theoretical foundation is a meaningful methodological contribution.
- The complete system shows promising practical benefits rather than only an improvement to an isolated estimator. With one hyperparameter configuration, it learns across four MuJoCo tasks and shows strong sample efficiency on Walker2d and Humanoid relative to the tested model-free baselines. The main performance curves report five-seed statistics, and the evaluation considers both environment interactions and training time. These results support the usefulness of the overall approach within the evaluated setting.
Critical weaknesses
- The ablations do not isolate the benefit of state–action-dependent guidance. Guided λ varies both across state–action pairs and over training, whereas fixed λ does neither. Figure 7 shows that the guided effective horizon starts shorter and increases over training. Its advantage could therefore come from a simple short-to-long rollout schedule, without requiring different cutoffs for different state–action pairs. Matching this training trend with a state-independent schedule would test the paper's claim that the cutoff location matters.
- The experiments provide limited evidence of reliable gains and broad applicability. (1) Key component ablations use only one seed, including the matched four-critic SAC/neither control, while the main SAC baseline uses two critics. (2) Several MBPO curves stop before the stated 400k-step budget. (3) The comparison lacks recent representative methods from the same research setting. (4) Only four Gymnasium MuJoCo tasks are tested. Together, these gaps limit confidence in the stability of the gains, their relevance against contemporary methods, and the transfer of the shared hyperparameter setting across tasks.
- (Minor) Taking the minimum over all critics ties the degree of pessimism to ensemble size and can lead to excessive underestimation as more critics are added. Consequently, the ensemble-size ablation changes both uncertainty estimation and value aggregation, making its results difficult to interpret. A mean-minus-uncertainty bound [1] would allow separate control of pessimism. The gate also does not explicitly account for aleatoric uncertainty in stochastic-policy returns [1].
[1] Yu et al. Taming Aleatoric Impulse in Off-Policy Reinforcement Learning. ICML, 2026.
Questions for authors
- Can the authors compare guided λ with a state-action-independent schedule matched to its effective horizon over training? These controls would clarify whether the gains require state-action-dependent cutoffs.
- Can the authors provide multi-seed results for every curve in all three panels of Figure 3, extending all currently single-seed variants and reporting the seed counts and variability? Please also provide complete MBPO curves, a recent representative baseline, and results on a small number of additional tasks under comparable budgets. A focused extension would be sufficient.
- The theory establishes contraction under an exact model, but does not show whether the estimator reduces value-estimation error with learned dynamics. The reported returns and training losses also do not directly establish lower value-estimation bias. Can the authors provide either further theoretical analysis explaining when the estimator improves accuracy, or a focused empirical comparison of its bias against that of the critic alone? For example, Monte Carlo estimates of discounted soft returns from the same state–action pairs under a frozen policy could provide a reference for measuring overestimation and underestimation on an existing task.
2 comments
I thank the authors for the additional experiments and theoretical analysis, which address most of my concerns. However, the experimental scope remains limited, particularly in terms of task diversity and comparisons with recent methods. I therefore maintain my original rating.
We thank the reviewer for the detailed feedback, which we used to extend the ablations, extend the contraction rate proof, and sharpen the positioning of the paper.
Q1: state-action-independent schedule
Q2: multi-seed results, MBPO, baselines
Q3: accuracy under learned dynamics
Minor: pessimism and ensemble size
Proof (Q3)
Definitions
Sketch
Qtλ−Qπ=(1−λt)(Qθ,t−Qπ)+λtγ(Vλ(x^t+1)−Vπ(x^t+1))+λtγ(Vπ(x^t+1)−Vπ(xt+1))
et≤(1−λt)δ+λtγ(et+1+εt)