- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Dual-Source Anchor Drift Correction (DADC) for online RL. It builds attractive anchors via an advantage-weighted flow prior plus Q-guided Langevin refinement, and repulsive anchors from the lowest-critic-scored current-policy samples. Beyond direct Q maximization, it adds drift guidance toward attractive and away from repulsive anchors, while keeping single-pass actor inference at deployment.
Critical strengths
- The dual-source anchor design directly targets the underutilization of negative feedback, and ablations support the complementarity of the two branches.
- The method retains single-pass inference while achieving consistent superiority across 14 tasks.
Critical weaknesses
- The paper lacks rigorous theoretical analysis to theoretically justify the effectiveness of the proposed method.
- The proposed method introduces a large number of hyperparameters and lacks systematic hyperparameter sensitivity analysis, which poses a substantial obstacle to its practical application.
Questions for authors
- In the first paragraph of the Introduction, the sentence "These objectives assign lower weights to low-value actions but ..." Please explain this in detail: what guidance does it provide?
- The paper lacks rigorous theoretical analysis to theoretically prove the effectiveness of the proposed method.
- The proposed method introduces a large number of hyperparameters, but the paper lacks a systematic hyperparameter sensitivity analysis to analyze the sensitivity of these hyperparameters, such as the Langevin hyperparameters (η, number of steps, noise), the prior update frequency d_target, α, etc. At the same time, the proposed method introducing too many hyperparameters makes the method less practical, because it requires users to tune a large number of hyperparameters when using it.
- The DRL community has developed a large number of advanced methods to drive the policy to obtain higher return values, which shares the same ultimate goal as generative RL, while the non-generative RL baselines in the experiments are very few. Therefore, in addition to comparing with SOTA generative RL, 5 additional SOTA non-generative RL methods should also be compared.
- The experiments in the paper are almost entirely of the locomotion / whole-body control type, and indeed lack manipulation tasks.
- The baselines in the experiments are basically generative RL based on diffusion policies and flow models. Generative RL includes various types of methods based on GAN, Transformer, VAE, etc., and these are also designed to maximize long-term rewards. Therefore, it is suggested to additionally compare with SOTA generative RL methods of other types.
- The efficiency comparison is only conducted on a single Dog Run task against a single Flow RL, which is insufficient. It is suggested to compare runtime efficiency with more SOTA algorithms on more tasks to demonstrate the superiority of the proposed method.