acceptodds
Under review as a conference paper at ICLR 2027

Evaluating OOD Actions in Offline RL via Quantile Advantage

Abstract

Offline reinforcement learning must improve on recorded behavior while controlling errors in evaluating out-of-distribution (OOD) actions. We introduce Advantage-Based Diffusion Actor-Critic (ADAC), which compares candidate and behavior actions through their successor values. A learned dynamics model predicts these successors; a state-value function learned from offline transitions evaluates them. Subtracting a quantile of behavior-induced values from the candidate's successor value defines its quantile advantage. This signed signal corrects critic targets, guiding a diffusion actor alongside behavior cloning. The comparison covers in-distribution and OOD actions; its accuracy depends on the learned dynamics and values. We characterize reference-value updates and the augmented fixed-policy evaluation operator. ADAC achieves the highest reported mean on 15 of 20 D4RL tasks. Four-domain component ablations support the correction's contribution, and a 15-task quantile study examines its reference threshold. PointMaze illustrates shorter routes beyond the recorded patterns. Code: https://anonymous.4open.science/r/adac-14D0.

Primary area:
reinforcement learning
Keywords:
Offline Reinforcement Learning, Out-of-Distribution Actions, Advantage Function, Diffusion Policies, Actor-Critic Methods

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QE1cOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Advantage-Based Diffusion Actor-Critic (ADAC), a method for offline reinforcement learning that evaluates candidate actions by comparing their predicted successor state values with a reference distribution derived from behavior actions. Specifically, the method uses a learned dynamics model to predict successor states, a state-value function to evaluate them, and a quantile of the values induced by behavior actions to construct a quantile advantage. This signal is incorporated into critic target updates, while the actor is parameterized by a diffusion model and trained using behavior cloning with critic guidance. The paper characterizes the fixed point of expectile-based value estimation and analyzes how the proposed advantage correction modifies the policy evaluation objective. Experiments on D4RL benchmarks and a PointMaze navigation task demonstrate competitive performance. Ablation studies further assess the contribution of the proposed correction and the effect of different quantile thresholds.

Critical strengths
  1. Instead of conservative penalties on OOD actions, the paper evaluates candidate actions relative to a reference distribution induced by the behavior actions. This provides a conceptually interesting way to use offline data both for behavior cloning and as a reference for action evaluation.
  2. The experiments cover 20 D4RL tasks across Gym, AntMaze, Adroit, and Kitchen. The paper also includes component ablations and a quantile sensitivity study, which provide evidence that the proposed correction contributes to performance and that the choice of reference threshold matters.
  3. The paper analyzes the fixed point of expectile-based value estimation and establishes contraction and perturbation bounds for the corrected policy evaluation operator. These results help characterize the mathematical properties of the proposed method.
Critical weaknesses
  1. While the paper is motivated by evaluating OOD actions, the proposed method does not explicitly distinguish actions based on their distributional status. Without a concrete metric for detecting OOD actions, it is difficult to determine whether the observed improvements result from more reliable evaluation of OOD actions or simply from improved value estimation among in-distribution candidates.
  2. The PointMaze results show that ADAC can produce shorter routes absent from the dataset, but this alone does not validate the central claim that the quantile advantage enables reliable evaluation of OOD actions. As the authors acknowledge, new paths can arise from stitching in-distribution actions rather than from true OOD actions. The paper also does not establish whether the actions along these shortcuts receive positive quantile advantages. Consequently, the observed shortcuts may be partly attributable to the diffusion policy’s inherent stitching ability, rather than to the proposed advantage correction itself.
  3. The quantile advantage in Eq. (4) depends on a learned dynamics model PψP_\psi and value model VφV_\varphi, both trained exclusively on in-distribution data. Prediction on OOD actions and OOD states may therefore require extrapolation beyond the training distribution, potentially leading to unreliable advantage estimates and misleading critic updates.
  4. Sensitivity to the quantile threshold κ\kappa. The optimal value of κ\kappa varies considerably across tasks, with certain settings causing performance collapse. This sensitivity raises concerns about the robustness of ADAC and the extent to which it can be applied without task-specific tuning.
Questions for authors
  1. Could the authors provide a direct analysis of the quantile advantage distributions for in-distribution and OOD actions, further distinguishing beneficial from detrimental OOD actions? In particular, how does A(a∣s)A(a \mid s) correlate with an explicit OOD detection metric, and to what extent can it distinguish beneficial OOD actions from detrimental ones?
  2. How can the algorithm trust the predicted next state and the corresponding value estimation V(s’) when it comes from an OOD action that the dynamics model has never seen? Have the authors considered using uncertainty estimates (e.g., via an ensemble) to assess prediction reliability?
  3. How do dataset characteristics, such as data quality and reward density, affect the sensitivity of ADAC to the quantile threshold κ\kappa? Is there a more adaptive way to set this threshold?
  4. In Table 3, ADAC adopts a deep 16-block ResNet critic for the majority of the tasks, while most standard baselines (including the original DQL) use shallow 3-layer MLPs. Can the authors provide an ablation to isolate how much of the performance gain originates from the increased model capacity versus the quantile advantage correction itself?
Reviewer_iQ8hOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes a new method to better evaluate and use OOD actions in offline learning. Instead of avoiding OOD actions, this paper suggests to evaluate OOD action by its successor, using a one-step dynamics model and value function learnt from offline dataset. It use the k-quantile of the behavior actions' successor values as baseline, and adds a positive or negative bonus to the critic target of DQL, depending on whether the candidate action's successor value is higher than the baseline, while keeping the rest of DQL unchanged. They also give four propositions for their method. In experiments, they include 20 D4RL tasks, design a PointMaze task, show ablation with and without the correction, and a k sweep, and the paper reports that their method outperforms other baselines.

Critical strengths
  1. The question and idea are worth doing. OOD actions could lead to better continuation in offline learning, how to better use it rather than preventing all of them is worth doing. The idea of using successor state to decide the OOD action is interesting.
  2. The experiments include many baseline models, as well as the designed PointMaze specifically shows the intended behavior.
Critical weaknesses
  1. The main claim depends on the learnt one-step dynamics model, but no test to show it's accurate. If the dynamics model is trained on the same offline dataset, its prediction on OOD actions should still be a problem. For example, previous methods may deal with OOD in Q(s,a), now this setup change to deal with OOD in dynamics model. Why would this be better?
  2. The full paper talks about OOD, but the paper doesn't show which actions are actually OOD. If the main claim is that ADAC evaluates and uses OOD actions, the paper should show how many OOD actions have been used in real cases.
Questions for authors
  1. Could you measure the prediction error of dynamics model on OOD actions, and check if the predicted successors stay close to the offline dataset states? Can I say the methods works mainly when the successor is not OOD, so that value function is reliable?
  2. In PointMaze, how can we know the shorter routes are built from truly OOD actions, or from in-distribution actions combined together?
  3. In PointMaze, it seems both DQL and ADAC could find short trajectories, just ADAC could find higher ratio of short trajectories. Can you confirm both models have been trained well, like would DQL keep improving with more training? and can you confirm both models uses same setup except the correction? The same question also applies to D4RL, for example, ADAC uses a ResNet critic in 14 of 20 tasks, did DQL use the same critic or how does DQL perform with the same critic?
  4. If Fig 2 is the expected output, can you use the real experiments data to generate something similar to Fig 2?
  5. Are the four propositions known results or are they new?

Minor comment:

  1. If you use blue in the fig 2, it's better to use the same color for the bar nearby
  2. The first paragraph in introduction shows literature about how other offline RL deal with OOD data, can you also include some most updated literature?
Reviewer_kBnXOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Advantage-Based Diffusion Actor-Critic (ADAC) for evaluating candidate actions, including out-of-distribution (OOD) actions, in offline reinforcement learning. A learned one-step dynamics model predicts successor states, and a separately learned state-value function evaluates their continuations. Comparing the candidate's successor value with a quantile of behavior-induced successor values defines its quantile advantage. This signal corrects critic targets and guides a diffusion actor through critic learning. The method is evaluated on 20 D4RL tasks and in PointMaze.

Critical strengths

Using behavior-induced successor values as a reference for comparing candidate actions is well motivated; the authors report that ADAC achieves the highest reported mean on 15 of 20 D4RL tasks and learns shorter routes beyond the recorded patterns in PointMaze.

Critical weaknesses
  1. Limited direct validation of OOD action evaluation. ADAC evaluates candidate actions using a learned dynamics model and a reference state-value function. For actions with limited data support, both components may produce unreliable estimates. The experiments mainly demonstrate improvements in policy returns, without directly validating the accuracy of successor-state predictions or the reliability of the quantile advantage signal. New routes in PointMaze can also be composed of behavior-supported actions and therefore do not, by themselves, demonstrate reliable evaluation of OOD actions. A direct empirical link between the performance gains and the proposed OOD action evaluation mechanism is still missing.

  2. Computational cost is not separated from implementation changes. The paper reports a 2x training speedup and a 1.64x inference speedup over the original DQL implementation, but the comparison includes a JAX reimplementation. ADAC introduces additional behavior-action sampling, successor-state prediction, and reference-value evaluation during training, yet these costs are not quantified separately. The total runtime, including all auxiliary pretraining, is also not fully reported. These comparisons therefore do not clearly establish the source of the speedups or the trade-off between performance gains and the computational cost of the correction.

  3. Missing direct comparison with ReFORM. The introduction states that the proposed approach complements ReFORM by using predicted successor values to compare candidate actions with behavior actions. However, the experimental evaluation does not include ReFORM. A direct comparison on common tasks under the same evaluation protocol is missing, leaving ADAC's empirical performance relative to this related method unclear.

Questions for authors
  1. Could the authors provide more direct empirical evidence supporting the reliability of the proposed method for evaluating OOD actions, and explain how estimation errors in the dynamics model and the reference state-value function affect this reliability?
  2. Could the authors compare running times with and without the correction using the same implementation framework, networks, and sampling settings, and report the total computational cost including all auxiliary model pretraining?
  3. Could the authors provide a direct comparison with ReFORM on the same tasks and under the same evaluation protocol to clarify the empirical performance of the two methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.