- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Noise-conditioned Q-Adjoint Anchoring (NQA) for offline-to-online reinforcement learning. It combines Q-Adjoint Anchoring, which uses adjoint-based critic guidance to train a single-step action sampler, with Noise-conditioned Q-Chunking, which connects long-horizon value learning to short-horizon policy execution. The reported OGBench results show substantial performance improvements over the evaluated baselines while maintaining efficient action generation.
Critical strengths
-
Well-motivated design addressing complementary challenges. The paper addresses an important trade-off between policy expressiveness, computational efficiency, and long-horizon value learning. Q-Adjoint Anchoring brings adjoint-based critic guidance into the training of a single-step action sampler, while Noise-conditioned Q-Chunking connects longer value-learning horizons to shorter policy execution. These components form a coherent approach to improving action selection while retaining efficient action generation and frequent feedback from the environment.
-
Broad evaluation with encouraging performance gains. The evaluation covers 65 OGBench tasks and eight relevant baselines, with eight random seeds for the main results. Under the reported experimental protocol, NQA achieves an overall online success rate of 76%, compared with 54% for the strongest evaluated baseline. The improvements on challenging humanoid navigation tasks are particularly encouraging and support the practical value of the proposed approach.
-
Useful supporting analyses and explicit theoretical scope. The paper supplements the main results with component ablations and sensitivity analyses for the regularization strength, adjoint clipping, expectile coefficient, and policy execution horizon. These studies help assess the design choices and their dependence on hyperparameters. The appendix also explicitly discusses the assumptions and limitations of the theoretical results, including the distinction between idealized objectives and practical updates.
Critical weaknesses
-
Limited matched comparisons at larger critic horizons. In the main comparison (Table 1), NQA, DQC, and CGQ are evaluated at the same critic horizon of . The extended-horizon study in Figure 7 (left) and Table 2 demonstrates that NQA benefits from increasing , but does not directly test whether DQC or CGQ would benefit similarly. The published LQL comparison in Appendix C.1 is encouraging, with NQA reporting higher success using fewer training steps, although the critic horizons differ. A direct comparison with DQC or CGQ at a larger matched horizon would strengthen the evidence for NQA's relative performance in this setting.
-
Limited task coverage for attributing the gains to individual components. The complete component comparison in Figure 7 (right) is conducted on puzzle-4x6-play-task1, while the comparisons with alternative actors in Figure 8 are conducted on humanoidmaze-medium-navigate-task1. These experiments provide useful evidence, but each comparison covers only one task. Consequently, the contributions of QAA, NQC, and their interaction remain less well established on humanoidmaze-large and humanoidmaze-giant, where the main results show particularly large gains. Broader component studies would make the explanation of these gains more convincing.
Questions for authors
- Comparison at a larger matched horizon. Could the authors compare NQA with DQC or CGQ on humanoidmaze-giant at one representative critic horizon , using the same dataset and comparable offline and online training budgets, while keeping the policy execution horizon fixed? This would help determine whether NQA's relative advantage persists when the competing methods also receive longer value-learning horizons.
3 comments
Response to authors
Thanks for the additional experiments. They largely address my main concerns. One quick question: Table 2 reports four seeds, while the rebuttal mentions three. How many seeds did you use for each method in the matched-horizon comparisons?
Dear reviewer 5ez2,
We are more than happy to hear that our rebuttal has addressed your concerns,
and we appreciate your further interest in our work.
In our submission, we used 8 seeds to collect the results in Figure 2 (Left), Figure 7 (Right), and Tables 1 and 12. All the other Figures and Tables use 4 seeds in the paper.
Thank you, and please let us know if there are any other points that you are curious of.
We additionally compared NQA, DQC, and CGQ with H∈{10,25}. Due to computational constraints, we focused on the
humanoidmaze-giant-navigate-100Mtasks from Table 12 (H=5) on which either CGQ or DQC achieved non-zero success rates.humanoidmaze-giant-navigate-100M-task1:Neither DQC nor CGQ showed any performance improvements across the three horizon values.
humanoidmaze-giant-navigate-100M-task2:We found that CGQ does not improve with larger values of H.
humanoidmaze-giant-navigate-100M-task5:We observed that while DQC improves with larger values of H, it still underperforms NQA.
Although these results may not fully capture the scaling potential of each method, we believe they further demonstrate NQA’s superior performance.
We further extended our second ablation study. Specifically, we evaluated how QAA and NQC individually contribute to performance improvements across 11 additional OGBench tasks.
humanoidmaze-large-navigate(5 tasks):cube-triple-play(5 tasks):humanoidmaze-giant-navigate-100M-task1(default task):We found that QAM+NQC also performs well on
humanoidmazetasks, highlighting the benefit of NQC, particularly on long-horizon and sparse-reward tasks. Furthermore, comparing QAM+DQC and QAA+DQC, we observed that QAA+DQC performs better, indicating that QAA improves not only computational efficiency but also task performance due to its enhanced stability (please refer to A.3 in our response to Reviewer v1Lf).We thank you for your valuable feedback and hope that our rebuttal addresses your concerns.