acceptodds
Withdrawn from ICLR 2027

Aleatoric Impulse Damping via Direct Evidential Regression

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Off-policy reinforcement learning can suffer from aleatoric impulse, a transient surge in value overestimation when stochastic exploration interacts with limited data. Suppressing this impulse requires aleatoric value uncertainty, a higher-order uncertainty of the learned value estimate itself. However, conventional distributional reinforcement learning models only lower-order aleatoric return uncertainty arising from the variability of return samples. Existing methods either overlook this distinction or heuristically construct value uncertainty from return uncertainty through scaling and annealing, leading to hyperparameter sensitivity and loss of state–action dependence. To address this issue, we propose **D**irect **E**videnti**A**l **R**egression (**DEAR**), a general framework for directly estimating the magnitude of higher-order uncertainty in parametric estimators. DEAR leverages temporal prediction variation and an exponential moving average anchor to learn this uncertainty. Building upon DEAR, we develop the **A**leatoric **I**mpulse **D**amping via Direct **E**vidential **R**egression (**AIDER**) algorithm, which augments a distributional critic with an evidential head to directly estimate value estimation uncertainty. Experiments on continuous-control benchmarks show that AIDER achieves state-of-the-art performance.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, Evidential learning, Aleatoric uncertainty, Value overestimation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 2 reviews, above 0% of the others.
Reviewer_ZwHgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce a method to estimate state-action higher-order aleatoric uncertainty from the variance of SGD iterates around an EMA anchor, with the goal of improving upon AID (Yu et al., 2026)'s global scaling, which dampens early-training overestimation.

Critical strengths
  • The approach is relatively clear and straightforward.
Critical weaknesses
  • The authors are eager to present their approach as the first heuristic-free method (the term is used 13 times in the paper), yet it merely shifts the heuristic of prior work to other parts of the pipeline: e.g., β\beta is a heuristic, and η\eta, BB, and τ\tau scale the surrogate's level, just as Amini et al.'s λ\lambda does.
  • Discrepancy between claim and theory. Theorem 1 shows that the surrogate converges to an SGD noise floor, while the target cross-data variance vanishes; i.e., DEAR does not estimate the absolute magnitude of higher-order uncertainty.
  • The implemented method is not covered by the theory. The residual between EMA targets used in the method is not covered by the theory.
  • The claim of state-of-the-art performance is unjustified. The authors report only three seeds and omit many baselines, such as TQC (Kuznetsov et al., 2020), REDQ (Chen et al., 2021), and SimBaV2 (Lee et al., 2025).

(Kuznetsov et al., 2020), Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics
(Chen et al., 2021), Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
(Lee et al., 2025), Hyperspherical Normalization for Scalable Deep Reinforcement Learning

Questions for authors
  1. (36) shows that the surrogate converges to a value >0> 0, while the target vanishes. Why is DEAR considered an accurate estimate of the absolute magnitude?
  2. How does DEAR's dependence on η\eta, BB, and τ\tau differ from its dependence on λ\lambda, which the paper considers heuristic?
  3. How does DEAR differ from prior work such as Mandt et al. (2017) and SWAG (Maddox et al., 2019), which already estimate uncertainty from SGD iterates?

(Mandt et al., 2017), Stochastic Gradient Descent as Approximate Bayesian Inference
(Maddox et al., 2019), A Simple Baseline for Bayesian Uncertainty in Deep Learning

Reviewer_JpVeOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper discusses uncertainty estimation in entropy-regularized reinforcement learning using deep evidential regression. It describes a Normal–Inverse-Gamma model for representing uncertainty in the parameters of a Gaussian likelihood and distinguishes lower-order from higher-order uncertainty.

Critical strengths

The paper addresses an important and interesting research problem. This topic has practical relevance for developing more reliable learning and decision-making methods.

Critical weaknesses

The notation and exposition need improvement. In Section II, γ\gamma and α\alpha are each used for two different quantities without sufficiently distinguishing their roles. The meaning of ZZ and several variance-related expressions is also unclear to me. These issues make it difficult to follow the formulation and subsequent arguments. The authors should use distinct symbols for different quantities, provide explicit definitions at first use, and explain the relationships among the uncertainty measures.

At a minimum, clear and consistent notation is necessary for readers to understand the work correctly.

Questions for authors

Could the authors use distinct symbols for the NIG predictive mean and the MDP discount factor, and for the NIG shape parameter and the entropy temperature?

Could the authors explicitly define ZZ when it first appears and explain its role in the formulation?

Could the authors define each variance quantity, specify the source of randomness and any conditioning involved, and explain how the different uncertainty measures relate to one another?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.