acceptodds
Under review as a conference paper at ICLR 2027

Noise-conditioned Q-Adjoint Anchoring

Abstract

We propose Noise-conditioned Q-Adjoint Anchoring (NQA), an inference-efficient technique that advances the state-of-the-art in offline-to-online reinforcement learning (RL). Expressive RL approaches combining generative policies, distributional critics, and action chunking achieve strong performance, yet incur substantial computational overhead. For efficient inference, multi-step integration in diffusion and flow policies can be replaced with straight flow sampling, but existing approaches encounter two expressivity limitations during training. Specifically, they struggle to (1) directly leverage critic guidance along straight flow trajectories and (2) capture distributional returns while reducing bootstrapping bias. To address these limitations, NQA introduces two algorithmic innovations. First, for policy learning, Q-Adjoint Anchoring applies Adjoint Matching to propagate critic gradients along straight flow trajectories while eliminating multi-step forward SDE integration. Second, for value learning, Noise-conditioned Q-Chunking estimates optimistic values within the return distribution while mitigating bias accumulation and maintaining a short policy execution horizon. In robotic locomotion and manipulation tasks, NQA achieves both high computational efficiency and strong task performance. To the best of our knowledge, NQA is the first method to achieve an average success rate above 90% on one of the most challenging tasks in OGBench, humanoidmaze-giant-navigate.

Primary area:
reinforcement learning
Keywords:
Offline-to-Online RL, Generative Policies, Distributional RL

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5ez2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Noise-conditioned Q-Adjoint Anchoring (NQA) for offline-to-online reinforcement learning. It combines Q-Adjoint Anchoring, which uses adjoint-based critic guidance to train a single-step action sampler, with Noise-conditioned Q-Chunking, which connects long-horizon value learning to short-horizon policy execution. The reported OGBench results show substantial performance improvements over the evaluated baselines while maintaining efficient action generation.

Critical strengths
  1. Well-motivated design addressing complementary challenges. The paper addresses an important trade-off between policy expressiveness, computational efficiency, and long-horizon value learning. Q-Adjoint Anchoring brings adjoint-based critic guidance into the training of a single-step action sampler, while Noise-conditioned Q-Chunking connects longer value-learning horizons to shorter policy execution. These components form a coherent approach to improving action selection while retaining efficient action generation and frequent feedback from the environment.

  2. Broad evaluation with encouraging performance gains. The evaluation covers 65 OGBench tasks and eight relevant baselines, with eight random seeds for the main results. Under the reported experimental protocol, NQA achieves an overall online success rate of 76%, compared with 54% for the strongest evaluated baseline. The improvements on challenging humanoid navigation tasks are particularly encouraging and support the practical value of the proposed approach.

  3. Useful supporting analyses and explicit theoretical scope. The paper supplements the main results with component ablations and sensitivity analyses for the regularization strength, adjoint clipping, expectile coefficient, and policy execution horizon. These studies help assess the design choices and their dependence on hyperparameters. The appendix also explicitly discusses the assumptions and limitations of the theoretical results, including the distinction between idealized objectives and practical updates.

Critical weaknesses
  1. Limited matched comparisons at larger critic horizons. In the main comparison (Table 1), NQA, DQC, and CGQ are evaluated at the same critic horizon of H=5H=5. The extended-horizon study in Figure 7 (left) and Table 2 demonstrates that NQA benefits from increasing HH, but does not directly test whether DQC or CGQ would benefit similarly. The published LQL comparison in Appendix C.1 is encouraging, with NQA reporting higher success using fewer training steps, although the critic horizons differ. A direct comparison with DQC or CGQ at a larger matched horizon would strengthen the evidence for NQA's relative performance in this setting.

  2. Limited task coverage for attributing the gains to individual components. The complete component comparison in Figure 7 (right) is conducted on puzzle-4x6-play-task1, while the comparisons with alternative actors in Figure 8 are conducted on humanoidmaze-medium-navigate-task1. These experiments provide useful evidence, but each comparison covers only one task. Consequently, the contributions of QAA, NQC, and their interaction remain less well established on humanoidmaze-large and humanoidmaze-giant, where the main results show particularly large gains. Broader component studies would make the explanation of these gains more convincing.

Questions for authors
  1. Comparison at a larger matched horizon. Could the authors compare NQA with DQC or CGQ on humanoidmaze-giant at one representative critic horizon H>5H>5, using the same dataset and comparable offline and online training budgets, while keeping the policy execution horizon fixed? This would help determine whether NQA's relative advantage persists when the competing methods also receive longer value-learning horizons.
3 comments

W.1. (Q.1.) Limited matched comparisons at larger critic horizons (H>5H>5).

A.1. Across three additional tasks evaluated at $H\in { 10,25 } $, we observed that while CGQ does not benefit from a larger HH, DQC does. Notably, NQA exhibits the greatest improvement and achieves the highest overall performance.

We additionally compared NQA, DQC, and CGQ with H∈{10,25}H \in \{10, 25\}. Due to computational constraints, we focused on the humanoidmaze-giant-navigate-100M tasks from Table 12 (H=5H=5) on which either CGQ or DQC achieved non-zero success rates.

humanoidmaze-giant-navigate-100M-task1:

task 1DQCCGQNQA
H=50→00→225→72
H=101→00→085→84
H=250→00→091→96

Neither DQC nor CGQ showed any performance improvements across the three horizon values.

humanoidmaze-giant-navigate-100M-task2:

task 2CGQNQA
H=50→6435→92
H=100→094→98
H=250→099→99

We found that CGQ does not improve with larger values of HH.

humanoidmaze-giant-navigate-100M-task5:

task 5DQCNQA
H=561→2379→87
H=1066→4799→99
H=2568→7096→99

We observed that while DQC improves with larger values of HH, it still underperforms NQA.

Although these results may not fully capture the scaling potential of each method, we believe they further demonstrate NQA’s superior performance.


W.2. Limited task coverage for attributing the gains to individual components.

A.2. Experiments across 11 additional tasks indicate that NQC consistently improves performance. QAA also outperforms QAM on average, while offering an even greater advantage in computational efficiency.

We further extended our second ablation study. Specifically, we evaluated how QAA and NQC individually contribute to performance improvements across 11 additional OGBench tasks.

humanoidmaze-large-navigate(5 tasks):

QAM+DQCQAM+NQCQAA+DQCNQA (=QAA+NQC)
task16→065→8139→269→85
task23→00→1413→330→52
task33→091→9869→4987→99
task49→074→9052→5451→79
task510→077→9353→5747→57
avg6→061→7545→3357→74

cube-triple-play(5 tasks):

QAM+DQCQAM+NQCQAA+DQCNQA (=QAA+NQC)
task11→07→990→033→100
task20→01→650→04→85
task30→01→220→08→82
task40→00→250→03→82
task50→00→10→01→72
avg0→02→420→010→84

humanoidmaze-giant-navigate-100M-task1(default task):

QAM+DQCQAM+NQCQAA+DQCQAA+NQC (=NQA)
task18→05→8040→5925→72

We found that QAM+NQC also performs well on humanoidmaze tasks, highlighting the benefit of NQC, particularly on long-horizon and sparse-reward tasks. Furthermore, comparing QAM+DQC and QAA+DQC, we observed that QAA+DQC performs better, indicating that QAA improves not only computational efficiency but also task performance due to its enhanced stability (please refer to A.3 in our response to Reviewer v1Lf).


We thank you for your valuable feedback and hope that our rebuttal addresses your concerns.

Reviewer_5ez2OpenReview ↗

Response to authors

Thanks for the additional experiments. They largely address my main concerns. One quick question: Table 2 reports four seeds, while the rebuttal mentions three. How many seeds did you use for each method in the matched-horizon comparisons?

Dear reviewer 5ez2,
We are more than happy to hear that our rebuttal has addressed your concerns, and we appreciate your further interest in our work.

In our submission, we used 8 seeds to collect the results in Figure 2 (Left), Figure 7 (Right), and Tables 1 and 12. All the other Figures and Tables use 4 seeds in the paper.

Thank you, and please let us know if there are any other points that you are curious of.

Reviewer_v1LfOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes Noise-conditioned Q-Adjoint Anchoring (NQA) for efficient offline-to-online reinforcement learning. The method combines two components.

First, Q-Adjoint Anchoring (QAA) extends Q-Adjoint Matching (QAM) to one-step flow policies. It computes adjoints along straight flow trajectories, avoiding the expensive forward SDE integration required by QAM.

Second, Noise-conditioned Q-Chunking (NQC) combines noise-conditioned Q-learning with decoupled action chunking. It trains a critic on longer action sequences while allowing the policy to execute shorter chunks.

Experiments on OGBench navigation and manipulation tasks show substantial improvements over existing methods in both performance and computational efficiency. Ablations also demonstrate the benefits of the two components.

Critical strengths
  1. Efficient policy optimization using Q-gradients. QAA provides an interesting way to use adjoint-based policy optimization without expensive forward SDE integration. By applying adjoint guidance directly to straight flow trajectories, the method retains information from the intermediate generation process while supporting one-step policy inference. This is particularly useful for robotic control, where inference speed is important.

  2. Strong experimental results. NQA achieves substantial improvements over existing offline-to-online RL methods, especially on challenging humanoid navigation tasks. The paper evaluates both performance and computational efficiency, and includes ablations of the policy and critic components. The results demonstrate that the proposed combination is effective in practice.

Critical weaknesses
  1. The individual contributions and benefits of combining QAA and NQC need clearer justification. Both components build on existing methods. QAA extends QAM and Flow Anchoring by applying adjoint guidance directly to straight flow trajectories. NQC combines noise-conditioned Q-learning with decoupled Q-chunking. While these are useful extensions, it is not entirely clear what makes their combination particularly effective. QAA can also work with a standard DQC critic, and NQC can improve other policy-learning methods, as shown in Appendix C.2. This suggests that the two components may provide largely independent improvements. The combination ablation in Figure 7 is helpful, but it is conducted on only one task. Although NQA performs best in that experiment, the results do not fully establish whether the two components offer additional benefits when used together, beyond their individual improvements. A broader analysis would help clarify the main contribution of NQA as a unified method.

  2. The reliability of the optimistic value estimates in NQC is not sufficiently examined. NQC trains its critic using action sequences of length H, but the policy only executes n actions before replanning, where n can be much smaller than H. To connect these two horizons, the method trains a shorter-horizon value function using an upper expectile of the critic's predictions. As a result, the value of a short action chunk is influenced by high-value continuations found in the dataset. However, the learned policy may not actually follow these continuations after replanning. For example, a recorded sequence may lead to a high return, but the policy may choose different actions after executing its first few steps. The short-horizon value could therefore be optimistic about what the policy can achieve. The experiments show that increasing H improves performance, but they do not directly establish whether the resulting value estimates accurately predict the returns of the executed policy.

Questions for authors
  1. Relationship between QAA and NQC: What makes NQC particularly suitable for QAA compared with other policy-learning methods? Could the authors extend the four-way comparison (QAM+DQC, QAM+NQC, QAA+DQC, and QAA+NQC) to more environments? This would help distinguish the individual improvements from any additional benefits of combining them.

  2. Accuracy of the optimistic critic: How accurately does the learned short-horizon value predict the return of the actual policy when H is much larger than n? Could the authors compare predicted values with returns from policy rollouts, particularly as H increases? This would help determine whether the longer critic horizon improves value accuracy or mainly provides more optimistic targets for policy improvement.

  3. Straight-flow approximation: QAA computes adjoints along straight interpolations rather than the controlled SDE trajectories used by QAM. How much does this approximation change the resulting adjoint guidance? Could the authors compare the gradients obtained from both approaches, or analyze when the straight-flow approximation is reliable?

4 comments

W.1. (Q.1.) The individual contributions and benefits of combining QAA and NQC need clearer justification.

A.1. Experiments across 11 additional tasks demonstrate that NQC consistently outperforms DQC, while QAA achieves higher average task success rates than QAM. Coupled with higher computational efficiency, these gains yield an even greater practical advantage.

QAA and NQC are two independent methods that we found to be the most effective for achieving both high task performance and high computational efficiency. Specifically, QAA bypasses multi-step forward SDE iterations during both training and inference, whereas NQC uses only two separate critics compared to three in DQC. Exploring improved value-learning methods for QAA or policy-learning methods for NQC remains a promising direction for future work.

We further extended our second ablation study to 11 tasks. Our analysis across five humanoidmaze-large-navigate tasks, five cube-triple-play tasks, and one humanoidmaze-giant-navigate task indicates that both NQC and QAA improve average task performance (please refer to A.2 in our response to Reviewer 5ez2). Notably, the clipping mechanism parameterized by λ\lambda substantially improves QAA's stability (please refer to A.3), which we believe explains why QAA outperforms QAM on average.


W.2. (Q.2.) How accurately does the learned short-horizon value predict returns when H is much larger than n?

A.2. Results on two navigation tasks with n=1n=1 show that a larger HH leads to more accurate value estimates.

We test whether a longer critic horizon HH improves the accuracy of NQC’s short-horizon value estimates. Specifically, we train NQA with a policy execution horizon of n=1n=1 and a critic horizon H∈{5,10,15}H \in \{5, 10, 15\}. At each evaluation interval (every 250k steps), we start from a fixed initial state s0s_0 and query the learned value Z(s0,a0)Z(s_0, a_0), where a0a_0 denotes the all-zero action. We then perform 32 rollouts of the current policy from s0s_0, where each rollout first executes a0a_0 and subsequently follows the policy, replanning at every step. We record the discounted return for each rollout, take the maximum across all 32 rollouts, and compute the absolute error relative to the initial estimated maximum value Z(s0,a0)Z(s_0, a_0), i.e., ∣Z(s0,a0)−maxreturn∣|Z(s_0, a_0)-\text{maxreturn}|.

antmaze-giant-navigate-task1 (default task):

training stepH=5H=10H=15
250k38.813.29.3
500k25.113.32.5
750k17.211.85.1
1M21.68.75.4
1.25M6.73.42.6
1.5M10.22.32.8
success rates5→853→8921→88

humanoidmaze-large-navigate-task1(default task):

training stepH=5H=10H=15
250k308.2180.7110.2
500k215.1140.595.0
750k182.3145.680.8
1M162.6144.9103.1
1.25M211.2149.675.9
1.5M195.1100.177.3
success rates71→8381→9577→96

We observe that the absolute error decreases as HH increases. We believe this provides quantitative evidence explaining why a larger HH leads to improved task performance.


Q.3. How much does the straight-flow approximation change the resulting adjoint guidance?

A.3. The deviation of the tilted velocity target ($ \hat v_\beta^*)fromthebehaviorvectorfield() from the behavior vector field (v_\beta$) increases under the straight-flow approximation. However, our clipping mechanism parameterized by λ\lambda mitigates this deviation, leading to improved task performance.

We believe comparing the average deviation of the tilted velocity target ($ \hat v_\beta^* ,obtainedfromtheadjoints)fromthebasevelocityfield(, obtained from the adjoints) from the base velocity field (v_\beta$) directly addresses your concern. To this end, we evaluated the mean squared error of these two across three settings that differ only in their forward policy SDE integration: (1) QAM-style, (2) QAA-style without $ \hat v_\beta^* $ clipping (λ=−1\lambda = -1), and (3) QAA-style with v^β∗\hat{v}_\beta^* clipping (λ≠−1\lambda \neq -1). When computing the adjoints, we used value estimates from a critic trained via DQC (κb=0.7,κd=0.5\kappa_b=0.7, \kappa_d=0.5) to completely decouple the effect of the policy.

humanoidmaze-large-navigate-task1 (default task):

training_stepsQAM+DQCQAA+DQC w/o clipping (λ=−1\lambda=-1)QAA+DQC w clipping (λ=3\lambda=3)
250k10.710.56.4
500k14.320.39.6
750k18.623.610.0
1M20.457.012.7
1.25M24.143.612.5
1.5M34.3213.513.1
success rates6→022→339→2

cube-double-play-task2 (default task):

training_stepsQAM+DQCQAA+DQC w/o clipping (λ=−1\lambda=-1)QAA+DQC w clipping (λ=1\lambda=1)
250k10.9121.70.9
500k2.85.70.3
750k3.06.30.2
1M2.42.40.2
1.25M6.944.20.5
1.5M26.230.50.6
success rates18→92→117→87

In both tasks, the straight-flow approximation without clipping increases the deviation $ \mathbb{E}[\Vert{}v_\beta - \hat{v}_\beta^*\Vert{}_2^2] $ compared to multi-step forward SDE integration. However, we observed that this increase does not necessarily degrade task performance. In contrast, the straight-flow approximation with clipping effectively mitigates this deviation and generally leads to stronger performance. We believe that λ\lambda-clipping in QAA enables stable and reliable NQA training, which in turn explains why QAA outperforms QAM on average.


We thank you for guiding us toward these findings and hope this addresses your concerns.

Thank you for the additional experiments and detailed responses.

The expanded four-way comparisons across 11 tasks are helpful. They show that QAA and NQC provide largely independent benefits. NQC brings substantial improvements on long-horizon tasks, while QAA can improve training stability and reduce the cost of policy updates. The results also clarify the roles of the two components.

The horizon experiments provide useful evidence that increasing H improves the value estimates. However, the evaluation uses the maximum return from 32 policy rollouts as the reference. This is a best-of-32 statistic, which is not necessarily the same as the expected return or the upper expectile predicted by the critic. I would encourage the authors to clarify why this is an appropriate comparison.

The clipping experiments also show that clipping reduces the deviation of the guided velocity target from the behavior vector field. This supports the role of clipping in stabilizing QAA. However, it does not directly show how close the adjoint guidance from QAA is to that from QAM. I would suggest clarifying this point or narrowing the corresponding claims.

Overall, the rebuttal addresses most of my concerns. Thank you for the additional experiments and clarifications.

Dear reviewer v1Lf,
We appreciate your interest and for sharing the remaining concerns about our work.


Q. The best-of-32 statistic is not necessarily the same as the expected return or the upper expectile predicted by the critic.

A. The best-of-32 statistic directly represents our theoretical target.

Theoretically, the contraction mappings of Noise-conditioned Q-Learning (NQL) and Noise-conditioned Q-Chunking (NQC) rely on the essential supremum, which directly aligns with the best-of-32 statistic. We employ expectile regression (κ=0.9\kappa=0.9) solely as a practical approximation to realize this theoretical target while mitigating empirical issues, such as value overestimation.

We think that comparing a specific expectile or quantile of the sampled returns is appropriate for evaluating traditional expectile- or quantile-based distributional RL methods, whereas comparing the mean of the sampled returns is suited for methods that rely on expected return estimates for policy learning.

Please let us know if there are any additional quantitative results you would like to see regarding the critic estimates.


Q. v^β∗\hat v_\beta^* clipping does not directly show how close the adjoint guidance from QAA is to that from QAM.

A. QAA maintains the exact adjoint calculation mechanism as QAM, but calculates the adjoints with different SDE trajectories (straight for QAA, and curved for QAM).

We would like to clarify that the adjoints calculated with QAA are the same as those of QAM. Therefore, if the meaning of "adjoint guidance" in your question is about the calculated adjoint values given a fixed SDE trajectory, they are the same.

However, QAA and QAM differ in how they sample each SDE trajectory for adjoint calculation. In other words, the intermediate action states xtx_t differ between QAA and QAM training. We believe comparing the absolute adjoint values at different xtx_t has little meaning, and this led us to compare $\mathbb{E}[||v_\beta-\hat v_\beta^*||^2_2] $ instead, which we thought fairly quantifies the relative adjoint guidance on different xtx_t.

Please let us know if there are any other quantitative results you would like to observe regarding the adjoints.


We appreciate your positive view once again and hope we addressed your remaining concerns.

Reviewer_jddtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Noise-Conditioned Q-Adjoint Anchoring (NQA), an efficient offline-to-online reinforcement learning framework. First, the paper demonstrates the benefits of adjoint matching and distributional returns through empirical experiments. Building on these findings, the authors introduce Q-Adjoint Anchoring and Noise-Conditioned Q-Chunking to leverage these advantages while improving inference efficiency. Experimental results show that NQA outperforms several baseline methods.

Critical strengths
  1. The paper provides theoretical justification for NQA, supported by empirical evidence.

  2. The paper systematically analyzes key components of offline-to-online RL with generative models, including action chunking, adjoint matching, and distributional returns.

Critical weaknesses
  1. Discrepancies in baseline performance. Several reproduced baseline scores are substantially lower than those reported in prior work. The authors should explain these discrepancies and clarify whether differences in experimental configurations contribute to the performance gaps. It would be valuable to evaluate NQA under the same experimental settings as prior work to determine whether it consistently outperforms strong baselines such as QAM.

  2. Limited scope of ablation studies. Several key comparisons are conducted on only a small number of tasks, making it difficult to assess the generality of the conclusions. It would be better that the authors can extend the ablation studies to a broader set of tasks and examine whether the conclusions remain consistent, particularly regarding the individual contributions of Q-Adjoint Anchoring and Noise-Conditioned Q-Chunking.

  3. Insufficient analysis of noise conditioning. The benefits of noise conditioning in Noise-Conditioned Q-Chunking are not sufficiently clear. Could the authors provide a controlled comparison between variants with and without noise conditioning? Quantitative results and a concrete case study would help illustrate when and why noise conditioning is beneficial. Furthermore, why does the critic need to receive the same noise sample as the policy? Additional theoretical justification or experimental evidence would help clarify the rationale for this coupling.

  4. Unclear motivation and mathematical presentation. The motivation and mathematical exposition could be organized more clearly to improve readability. In particular, some key concepts are introduced before the necessary background has been established. For example, the construction in Eq. (6) relies on a behavior flow trained using the objective in Eq. (11), but this objective is only presented later in the appendix. Introducing the relevant definitions and training objectives earlier would make the methodological development easier to follow.

  5. Insufficient distinction from prior work. Several components of the proposed framework, including adjoint matching, Q-chunking, flow anchoring, and noise-conditioned Q-learning, build on ideas introduced in prior work. Although the paper devotes considerable space to describing these components, it does not sufficiently emphasize what is novel about the proposed approach. The authors should more explicitly distinguish existing techniques from their own contributions, particularly the proposed anchoring mechanism and noise-conditioning strategy for Q chunking, and explain how their integration differs from prior methods.

Questions for authors

Please refer to the concerns and questions raised in the Weaknesses section.

1 comment

W.1. Several reproduced baseline scores are substantially lower than those reported in prior work.

A.1. We use a higher discount factor, a smaller critic ensemble, and no gradient clipping. We chose this setup to target longer effective horizon and isolate the core algorithmic implementation.

One of our main goals was to capture long-horizon returns, which requires a high discount factor. However, a higher discount factor generally leads to unstable training due to increased variance in the value estimates. Our experiments demonstrate that NQA remains stable even with a high discount factor while outperforming the reported results of QAM (Please refer to Table 13). Specifically, QAM uses a discount factor of 0.99 for OGBench puzzle/scene/cube/antmaze-large tasks and 0.995 for humanoidmaze/antmaze-giant tasks. In contrast, we use 0.999 for humanoidmaze and 100M-sized tasks, and 0.995 for all the other tasks.

Moreover, we removed several implementation heuristics. For example, the original QAM experiments use a critic ensemble size of 10 with pessimistic updates and apply max-norm gradient clipping. Because these components are orthogonal to the underlying mathematical formulation of each algorithm, we omitted them to isolate the core algorithmic differences.


W.2. The individual contributions of QAA and NQC are conducted on only a small number of tasks.

A.2. Across 11 additional tasks, we observed that NQC and QAA consistently improve task performance.

Please refer to A.2 in our response to Reviewer 5ez2 for full results. Furthermore, we found that NQC with a high critic horizon length HH estimates values with lower error relative to the sampled returns, while at the same time, QAA enhances training stability through its clipping mechanism (Please refer to A.2 and A.3 in our response to Reviewer v1Lf). We believe the improved task performance is largely attributable to these two factors.


W.3. Why is noise-conditioning beneficial?

A.3. Theoretically, it models distributional returns much more efficiently. Empirically, it leads to better task performance.

In the original FAN paper, the authors state that the primary motivation for NQL is to model distributional returns without multiple quantile or expectile samples. Similarly, we argue that such multi-sample tracking is inefficient because the optimal value changes dynamically during policy improvement.

Without noise conditioning, we believe the approach no longer models distributional returns. In Figures 4 and 8, we demonstrate the empirical advantage of the noise-conditioned critic over critics without noise conditioning.

Likewise, if the critic does not receive the same noise sample as the policy, it also fails to model distributional returns. In NQL/NQC, each critic noise input is intended to map to different possible return estimates, as in standard generative modeling. However, calculating value targets using independent policy noise inputs causes every critic noise input to map to the expected return. As a result, it no longer functions as a distributional critic.

Please refer to A.2 of our response to Reviewer v1Lf for additional details regarding this concern.


W.4. Eq. (6) relies on Eq. (11), which is only presented later in the appendix.

A.4. Preliminaries → Extended Preliminaries in Appendix A.

If you find it appropriate, we will replace the current preliminaries in the main paper with the expanded version from the appendix. Please let us know if any further details remain unclear.


W.5. Does not sufficiently emphasize what is novel.

A.5. We respectfully note that we explicitly highlighted NQA’s novelty on pages 2 and 6, and provided detailed comparisons with prior work in the Related Work section. However, if any specific distinctions remain unclear, we would be glad to emphasize them further in the revision.

We agree that our work builds upon substantial prior research, which requires careful effort to delineate clearly. To this end, we structured our presentation to highlight our main contributions through two core components: Q-Adjoint Anchoring (QAA) and Noise-Conditioned Q-Chunking (NQC). Specifically, our first key contribution is bypassing the forward SDE integration of policies trained with adjoints (QAA), and our second is capturing distributional returns via Q-Chunking while detaching the policy execution horizon to enable shorter execution (NQC).

We respectfully note that these explanations are presented on page 2 (as numbered items detailing QAA and NQC) as well as on page 6. Moreover, we explicitly discuss differences from prior work on page 2 (via numbered items outlining the limitations of existing methods) and in the Related Work section (in the concluding sentences of each paragraph).


We thank you once again for your valuable guidance and hope that our response addresses your concerns.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Letter of Appreciation

Dear Reviewers jddt, v1Lf, and 5ez2,

We sincerely appreciate your time and effort in reviewing our submission. Your guidance has greatly helped us understand the deeper scientific principles behind NQA. Regarding the additional results provided in this rebuttal, please note that we used three random seeds and followed the hyperparameters in Tables 6 and 7 unless otherwise specified. We would be grateful for any further discussion to help us uncover deeper insights into our work.

AuthorsOpenReview ↗

Revision including the reviewers' concerns

Dear Reviewers jddt, v1Lf, and 5ez2,
We appreciate the time and effort you put into reviewing our submission.

We revised the manuscript to address the reviewers' main concerns.

  • First, we included the "Flow Matching" part in the Preliminaries section on the main page and eliminated the Extended Preliminaries section in the Appendix. (Reviewer jddt)
  • Second, we added Figure 8, which includes additional experiments that compare QAM+DQC, QAM+NQC, QAA+DQC, and NQA. (Reviewers jddt, v1Lf, 5ez2)
  • Third, we incorporated the "6.3 Further Analyses" section on page 10. This includes the value estimate error test (Reviewers jddt, v1Lf) and the adjoint guidance-related experiments (Reviewer v1Lf) during the rebuttal.
  • Fourth, we moved the Future Work part that previously existed in the Conclusion into Appendix A due to page limits.

We decided not to include the comparison of DQC, CGQ, and NQA at higher HH values, as it could harm CGQ's reputation.

Thank you for your positive views on our work, and we hope the revision addresses the reviewers' remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.