- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Advantage-Based Diffusion Actor-Critic (ADAC), a method for offline reinforcement learning that evaluates candidate actions by comparing their predicted successor state values with a reference distribution derived from behavior actions. Specifically, the method uses a learned dynamics model to predict successor states, a state-value function to evaluate them, and a quantile of the values induced by behavior actions to construct a quantile advantage. This signal is incorporated into critic target updates, while the actor is parameterized by a diffusion model and trained using behavior cloning with critic guidance. The paper characterizes the fixed point of expectile-based value estimation and analyzes how the proposed advantage correction modifies the policy evaluation objective. Experiments on D4RL benchmarks and a PointMaze navigation task demonstrate competitive performance. Ablation studies further assess the contribution of the proposed correction and the effect of different quantile thresholds.
Critical strengths
- Instead of conservative penalties on OOD actions, the paper evaluates candidate actions relative to a reference distribution induced by the behavior actions. This provides a conceptually interesting way to use offline data both for behavior cloning and as a reference for action evaluation.
- The experiments cover 20 D4RL tasks across Gym, AntMaze, Adroit, and Kitchen. The paper also includes component ablations and a quantile sensitivity study, which provide evidence that the proposed correction contributes to performance and that the choice of reference threshold matters.
- The paper analyzes the fixed point of expectile-based value estimation and establishes contraction and perturbation bounds for the corrected policy evaluation operator. These results help characterize the mathematical properties of the proposed method.
Critical weaknesses
- While the paper is motivated by evaluating OOD actions, the proposed method does not explicitly distinguish actions based on their distributional status. Without a concrete metric for detecting OOD actions, it is difficult to determine whether the observed improvements result from more reliable evaluation of OOD actions or simply from improved value estimation among in-distribution candidates.
- The PointMaze results show that ADAC can produce shorter routes absent from the dataset, but this alone does not validate the central claim that the quantile advantage enables reliable evaluation of OOD actions. As the authors acknowledge, new paths can arise from stitching in-distribution actions rather than from true OOD actions. The paper also does not establish whether the actions along these shortcuts receive positive quantile advantages. Consequently, the observed shortcuts may be partly attributable to the diffusion policy’s inherent stitching ability, rather than to the proposed advantage correction itself.
- The quantile advantage in Eq. (4) depends on a learned dynamics model and value model , both trained exclusively on in-distribution data. Prediction on OOD actions and OOD states may therefore require extrapolation beyond the training distribution, potentially leading to unreliable advantage estimates and misleading critic updates.
- Sensitivity to the quantile threshold . The optimal value of varies considerably across tasks, with certain settings causing performance collapse. This sensitivity raises concerns about the robustness of ADAC and the extent to which it can be applied without task-specific tuning.
Questions for authors
- Could the authors provide a direct analysis of the quantile advantage distributions for in-distribution and OOD actions, further distinguishing beneficial from detrimental OOD actions? In particular, how does correlate with an explicit OOD detection metric, and to what extent can it distinguish beneficial OOD actions from detrimental ones?
- How can the algorithm trust the predicted next state and the corresponding value estimation V(s’) when it comes from an OOD action that the dynamics model has never seen? Have the authors considered using uncertainty estimates (e.g., via an ensemble) to assess prediction reliability?
- How do dataset characteristics, such as data quality and reward density, affect the sensitivity of ADAC to the quantile threshold ? Is there a more adaptive way to set this threshold?
- In Table 3, ADAC adopts a deep 16-block ResNet critic for the majority of the tasks, while most standard baselines (including the original DQL) use shallow 3-layer MLPs. Can the authors provide an ablation to isolate how much of the performance gain originates from the increased model capacity versus the quantile advantage correction itself?