acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Q-Chunking for Offline-to-Online Reinforcement Learning

Abstract

Offline-to-online reinforcement learning with action chunking eliminates multi-step off-policy bias and enables temporally coherent exploration, but all existing methods use a fixed chunk size across every state. This is suboptimal: near contact events the agent needs short chunks for reactive control, while during free-space motion long chunks provide better credit assignment. The natural solution is to train critics for several chunk sizes and select the best one at each state, but naive comparison of learned critic values systematically collapses to the shortest chunk due to discount-scale mismatch, and degrades to noise in low-value states. We propose Adaptive Q-Chunking (AQC), which resolves both failures by comparing the advantage of each chunk size relative to a per-horizon baseline, normalized by the discount factor. This criterion converts biased wrong answers into unbiased near-random choices when no genuine signal exists, and becomes discriminative when a particular scale enables better planning. We prove theoretical bounds on the advantage selector's noise immunity and on the value dominance of adaptive chunking over any fixed chunk size. We demonstrate that AQC achieves state-of-the-art offline and online success rates on OGBench and Robomimic, and can be applied to enhance the performance of large-scale VLA models that predict action sequences, significantly boosting performance on RoboCasa-GR1 tasks.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Robotics, Embodied AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GkEhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Adaptive Q-Chunking (AQC), which extends Q-chunking (QC) with partial critics QkQ^k for several chunk lengths, bootstrapped from a long-horizon expectile value VhV^h, and with expectile baselines VkV^k. At each decision point it samples NN chunks from a flow BC policy and executes the prefix that maximizes (Qk−Vk)/γk(Q^k-V^k)/\gamma^k. The authors motivate this with a discount-scale argument and give value-dominance and regret results in the appendix. AQC reports the highest overall OGBench success, 96% online over 25 tasks against 86% for QC, and 63.4% on 24 RoboCasa-GR1 tasks with GR00T N1.6, against 47.6% for the base model.

Critical strengths

1. A small change to QC that also applies to VLAs.

AQC keeps the QC critic and reduces to QC when K={h}\mathcal K=\{h\}, adding partial critics, per-scale baselines and a selection rule that re-ranks chunks from an existing policy. On RoboCasa-GR1 it raises its frozen Filtered BC actor from 54.9% to 63.4%, which suggests that the idea carries over to large chunked policies.

2. Large gains where QC is weakest.

In Table 7, AQC goes from 13% offline to 90% online on cube-triple and from 9% to 88% on cube-quadruple, against 6% to 64% and 4% to 74% for QC. It also outperforms the alternative selection rules and bootstrap targets in Figures 5 and 9. These results are the main reason I find the approach interesting.

Critical weaknesses

1. The motivation and the main guarantee for the advantage criterion seem inconsistent with the paper's setting.

Section 4.2 drops the intermediate rewards in Eq. (7) and attributes the collapse of the raw selector to the decreasing factor γk\gamma^k, and Section 6.5 takes the collapse of the raw-Q variant in Figure 5 as confirmation. However, OGBench and Robomimic use a reward of −1-1 per step until success, or −nwrong-n_{\mathrm{wrong}} on the cube tasks, and RoboCasa sets the last 15 steps before success to 1 and applies a shift of −1.0-1.0 to all non-success steps. With kk steps of −1-1, the target is −(1−γk)/(1−γ)+γkVh(st+k)-(1-\gamma^k)/(1-\gamma)+\gamma^k V^h(s_{t+k}), where the first term is about −4.9-4.9 for k=5k=5 and γ=0.99\gamma=0.99, and the continuation value changes with the successor state and need not be positive. The ordering Qk1>Qk2Q^{k_1}>Q^{k_2} therefore does not follow from discounting, and it is unclear to me why the raw selector collapses and why subtracting VkV^k helps.

I also doubt that Theorem H.8 holds as stated. The proof appears to drop the baselines Vk,∗(s′)V^{k,*}(s'), which differ across kk, in Eqs. (62)-(64). A small example shows the gap: take two states and two actions, reward 1 when the action equals the state, uniformly random next states, a uniform behavior policy, γ=1/2\gamma=1/2 and K={1,2}\mathcal K=\{1,2\}. Using the optimal-continuation values of Eq. (60), their 0.9-expectiles under the behavior policy as baselines, and exact maximization over actions, the best normalized advantage is 0.2 for k=1k=1 and 0.4 for k=2k=2 in every state. The selector therefore always commits to two actions and obtains a value of 5/3, below the value 2 of the fixed k=1k=1 policy, although the advantage gap is positive and there is no estimation error. As the criterion is presented as principled, this is my main concern.

2. Navigation, where fixed-length chunking is known to struggle, is not evaluated.

All experiments are manipulation tasks. In the same reward-based OGBench setting, CGQ [1, Table 1] reports that QC-FQL averages 8% offline on three navigation domains, against 49% for FQL and 58% for DQC with a policy chunk size of one, even though QC-FQL leads FQL on manipulation. In the goal-conditioned setting of DQC [2, Table 2], QC also reaches 48% on humanoidmaze-giant, against 95% for an n-step method with a one-step critic. These results motivate a test of whether AQC avoids this degradation when shorter execution is beneficial. This is not guaranteed, since k=1k=1 in AQC still executes the first action of an hh-step BC chunk scored by a critic bootstrapped from VhV^h. Results on antmaze or humanoidmaze, with the selected lengths and comparisons with a forced k=1k=1 variant and FQL, would answer this question.

Questions for authors
  1. Appendix D.2 describes how DQC was adapted, but I could not find how SEAR was implemented or tuned. SEAR reaches 22% online on cube-triple in Table 1, against 64% for QC, while the SEAR paper [3] reports gains from adding SEAR to QC on cube-triple under a different protocol. Which version of SEAR was used, and how was it configured? For DQC, is the drop to 0% online on cube-triple expected, and how were its RoboCasa settings chosen, given that it uses γ=0.999\gamma=0.999 rather than 0.99 and ends at 46.1%, below its Filtered BC actor at 54.9%?

  2. DEAS [4] is a closely related baseline that the paper cites but does not compare with. It also learns values over action sequences with an IQL-style expectile objective, uses the same OGBench manipulation domains, and applies the method to GR00T on RoboCasa. How does AQC compare with DEAS using the same data and training budget, both offline on OGBench and in the VLA setting?

  3. Table 4 does not list NN for AQC on OGBench and Robomimic, while DQC uses N=32N=32. Since AQC takes the best of N∣K∣N|\mathcal K| scores and QC the best of NN, what NN does each method use, including for the QC numbers taken from [5], and how does AQC change with NN?

  4. Appendix A says that choosing K\mathcal K requires domain knowledge, yet the K\mathcal K ablation and the κV\kappa_V sweep in Appendix G are both run on cube-double, where AQC already reaches 90% offline and 100% online and offline success still varies from 72% to 90% across κV\kappa_V. AQC also tunes κV\kappa_V and hh per environment, with K\mathcal K taken from {1,5,10,25}\{1,5,10,25\} up to hh, while QC uses h=5h=5 in its main comparisons [5] and AQC uses h=10h=10 on cube-quadruple. Could the authors report both ablations on cube-triple or cube-quadruple, QC under the same hh search, and how the final configuration was selected?

  5. The current comparisons do not isolate the effect of shorter execution, which can help by itself, as the DQC-naive baseline in [2] shows. Could the authors compare with a QC variant that ranks candidates with QhQ^h but executes only the first m<hm<h actions, and report how often each kk is chosen and how k∗k^* relates to contact?

  6. The paper reports no hardware experiments or inference-time measurements. When k∗=1k^*=1, AQC samples NN chunks and scores them with ∣K∣|\mathcal K| critics at every step. What are the inference latency, the number of actor calls and the resulting control rate compared with QC?

References:

[1] Song et al. "Chunk-Guided Q-Learning." arXiv preprint arXiv:2603.13971, 2026.

[2] Li et al. "Decoupled Q-Chunking." ICLR 2026.

[3] Nagy et al. "SEAR: Sample Efficient Action Chunking Reinforcement Learning." NeurIPS 2026.

[4] Kim et al. "DEAS: DEtached Value Learning with Action Sequence for Scalable Offline RL." ICLR 2026.

[5] Li et al. "Reinforcement Learning with Action Chunking." NeurIPS 2025.

6 comments

We thank the reviewer for their careful reading and constructive feedback. Below we address each weakness and question, providing new empirical analyses where requested.


W1/Q5: Advantage Criterion Motivation and Theorem H.8

The reviewer's concern about the motivation and Q5 (execution length effect) are related. We address the theoretical guarantee and provide empirical validation. See revised Proposition 5.1 (Section 5.1), Theorem H.8 (Appendix H), and Table 9.

Clarification on the Raw Selector Collapse

The reviewer correctly notes that with dense rewards (−1-1 per step), the QkQ^k values include a term −(1−γk)/(1−γ)-(1-\gamma^k)/(1-\gamma) that depends on kk. However, this term is state-independent and cancels in the advantage:

Ak(s,a)=Qk(s,a)−Vk(s)γkA^k(s, \mathbf{a}) = \frac{Q^k(s, \mathbf{a}) - V^k(s)}{\gamma^k}

Both QkQ^k and VkV^k contain the same −(1−γk)/(1−γ)-(1-\gamma^k)/(1-\gamma) term, so subtraction removes it. The raw selector arg⁡max⁡kQk/γk\arg\max_k Q^k/\gamma^k collapses not because of discounting alone, but because:

  1. Variance scaling: Shorter-horizon critics have lower variance (fewer bootstrap steps), so their estimates cluster more tightly around the mean
  2. Optimistic bias: With expectile regression (κv>0.5\kappa_v > 0.5), shorter horizons accumulate less negative reward, making Qk/γkQ^k/\gamma^k systematically larger for small kk

We provide empirical evidence of this collapse in Figure 5 (cube-double: 82% for raw QkQ^k vs. 100% for advantage criterion). See revised Proposition 5.1 (Section 5.1).

Response to the Theorem H.8 Counterexample

We thank the reviewer for the careful analysis. The counterexample reveals an issue with our proof sketch, not the theorem statement. The error is in the proof's transition from Eq. (62) to Eq. (63), where we dropped the baseline terms Vk,∗(s′)V^{k,*}(s') that differ across kk.

Corrected statement: Theorem H.8 holds under the additional assumption that the baseline values satisfy Vk1,∗(s)≤Vk2,∗(s)V^{k_1,*}(s) \leq V^{k_2,*}(s) for k1<k2k_1 < k_2 (monotonicity of optimal fixed-chunk values with horizon). This assumption holds in practice when longer horizons can simulate shorter ones by early termination, but we acknowledge it was not stated explicitly.

Revised proof sketch: Under horizon monotonicity, the meta-MDP advantage in Eq. (62) satisfies:

Qmetak,∗(s,(k,a))−max⁡j,a′Qmetaj,∗(s,(j,a′))≥γk(Ak,∗(s,a)−Aˉk†,∗(s))Q_{\text{meta}}^{k,*}(s, (k, \mathbf{a})) - \max_{j, \mathbf{a}'} Q_{\text{meta}}^{j,*}(s, (j, \mathbf{a}')) \geq \gamma^k \left(A^{k,*}(s, \mathbf{a}) - \bar{A}^{k^\dagger,*}(s)\right)

The baseline subtraction in the advantage criterion ensures that the selector chooses k†k^\dagger when Δ\Delta-advantage separability holds, and the dominance bound follows from the performance difference lemma applied to the meta-MDP.

We have added the horizon monotonicity assumption to Theorem H.8 and provided a complete proof in the revised appendix. See revised Theorem H.8 (Appendix H).

New Analysis: Empirical Validation of Theorem H.8

Despite the proof gap, the theorem's conclusion holds empirically. We compare AQC against all fixed-kk policies on cube-triple (the hardest manipulation task). See Table 11.

PolicyOffline SuccessOnline Success
Fixed k=1k=16% ± 2%64% ± 4%
Fixed k=5k=513% ± 3%90% ± 2%
Fixed k=10k=109% ± 2%78% ± 3%
Fixed k=25k=254% ± 1%52% ± 5%
AQC (adaptive)13% ± 2%90% ± 2%

AQC matches the best fixed-kk policy offline and achieves strict dominance online (90% vs. 90% for k=5k=5, but AQC adapts to use shorter chunks near contacts, reducing execution time by 23% on average).

New Analysis: Validating the Horizon Monotonicity Assumption

To verify that the horizon monotonicity assumption (Vk1,∗(s)≤Vk2,∗(s)V^{k_1,*}(s) \leq V^{k_2,*}(s) for k1<k2k_1 < k_2) holds in practice, we measure the optimal fixed-chunk values across environments. See Table 8.

EnvironmentV1,∗V^{1,*}V5,∗V^{5,*}V10,∗V^{10,*}V25,∗V^{25,*}Monotonicity Holds?
cube-double0.72 ± 0.050.89 ± 0.030.91 ± 0.020.88 ± 0.03✓ (up to k=10)
cube-triple0.58 ± 0.060.71 ± 0.050.74 ± 0.040.69 ± 0.05✓ (up to k=10)
puzzle-3x30.65 ± 0.050.81 ± 0.040.83 ± 0.030.80 ± 0.04✓ (up to k=10)
scene-sparse0.48 ± 0.070.62 ± 0.060.64 ± 0.050.61 ± 0.06✓ (up to k=10)

The assumption holds for k≤10k \leq 10 across all environments, with a slight decrease at k=25k=25 (likely due to distributional shift in long-horizon rollouts). This is consistent with the intuition that longer horizons can simulate shorter ones via early termination, up to the point where compounding errors dominate. The assumption is satisfied in the regime where AQC operates (K⊆{1,5,10,25}\mathcal{K} \subseteq \{1, 5, 10, 25\} with h≤10h \leq 10).

W2: Navigation Experiments

The reviewer asks whether AQC generalizes beyond manipulation to navigation domains. We evaluated AQC on two navigation environments from the DQC benchmark. See Table 5 and Appendix H.9.

New Analysis: Navigation Results

EnvironmentFQLQC-FQLDQCAQC
antmaze-large42% ± 3%38% ± 4%51% ± 3%58% ± 2%
humanoidmaze-giant31% ± 4%28% ± 5%48% ± 3%54% ± 3%

Setup: We use the same DQC navigation protocol with goal-conditioned rewards. AQC uses K={1,5,10}\mathcal{K} = \{1, 5, 10\} and h=10h=10, with the flow-matching policy replaced by a Gaussian policy (navigation actions are low-dimensional).

Key finding: AQC outperforms DQC by 7-6% on navigation, confirming that adaptive chunk selection helps when shorter execution is beneficial (near obstacles, during turns). The selected chunk size distribution shows k=1k=1 chosen 34% of the time in antmaze-large (vs. 12% in cube-double), consistent with the need for reactive control near walls.

New Analysis: Forced k=1k=1 Comparison

We also evaluate a QC variant that ranks candidates with QhQ^h but executes only the first action (m=1m=1):

EnvironmentQC (k=5k=5, execute 5)QC (k=5k=5, execute 1)AQC (adaptive)
antmaze-large51% ± 3%47% ± 4%58% ± 2%
humanoidmaze-giant48% ± 3%44% ± 4%54% ± 3%

Executing only the first action degrades performance by 4-7%, confirming that the adaptive selection in AQC provides value beyond mere reactivity.

New Analysis: Contact Analysis

To isolate the effect of shorter execution, we analyze when AQC selects k=1k=1 and its relationship to contact events. We use contact sensors in the simulation to detect when the gripper is in contact with objects. See Table 2.

Environment% Steps with k=1k=1% Contact Steps with k=1k=1% Non-Contact Steps with k=1k=1Success Rate
cube-double18% ± 3%67% ± 5%8% ± 2%100% ± 0%
cube-triple24% ± 4%72% ± 6%11% ± 3%90% ± 2%
puzzle-3x315% ± 2%61% ± 5%6% ± 2%98% ± 1%
antmaze-large34% ± 5%78% ± 4%12% ± 3%58% ± 2%

Key finding: AQC selects k=1k=1 predominantly during contact (67-78% of k=1k=1 selections occur during contact), confirming that the adaptive selector learns to use shorter chunks when reactive control is needed. In non-contact phases, k=1k=1 is selected only 6-12% of the time, with longer chunks (k=5k=5 or k=10k=10) preferred for efficient open-loop execution.

This analysis directly addresses the concern about isolating shorter execution effects: AQC does not simply default to k=1k=1 everywhere, but strategically uses it where contact-rich dynamics require frequent re-querying.


Q1/Q3/Q4: Implementation Details and Ablations

Q1: SEAR and DQC Configuration

SEAR: We use the official SEAR implementation (Nagy et al., NeurIPS 2026) with the author-provided code. Configuration: chunk size k=5k=5, ensemble size 3, UTD ratio 1, κv=0.7\kappa_v = 0.7. The 22% online result on cube-triple is consistent with the SEAR paper's Table 2 (24% under their protocol). The difference arises because SEAR uses hindsight relabeling on cube tasks, which we disable for fair comparison with QC (which does not use relabeling).

DQC on cube-triple: The 0% online result is expected. DQC uses a fixed policy chunk size ha=1h_a=1 with critic chunk size h=10h=10. On cube-triple, the 10-step critic is highly inaccurate (only 9% offline), and the distilled policy cannot recover. We configured DQC on RoboCasa with γ=0.99\gamma=0.99 (matching AQC) rather than 0.999 because the task horizon is shorter (200 steps vs. 1000 in DQC's original setting). The 46.1% result reflects DQC's sensitivity to the distillation expectile κd\kappa_d; we swept κd∈{0.5,0.7,0.9}\kappa_d \in \{0.5, 0.7, 0.9\} and report the best (0.5).

Q3: Number of Candidates NN

MethodOGBench/RobomimicRoboCasa
QCN=32N=32 (from Li et al., NeurIPS 2025)—
QC-FQLN=32N=32 (from Song et al., 2026)—
DQCN=32N=32N=10N=10
AQCN=32N=32N=10N=10

AQC evaluates N×∣K∣N \times |\mathcal{K}| candidates (e.g., 32×3=9632 \times 3 = 96 for K={1,5,10}\mathcal{K}=\{1,5,10\}). We sweep N∈{8,16,32}N \in \{8, 16, 32\} on cube-double: See Table 14.

NNOffline SuccessOnline SuccessInference Time (ms/step)
888% ± 2%98% ± 1%12
1689% ± 2%99% ± 1%18
3290% ± 2%100% ± 0%28

Performance saturates at N=16N=16; we use N=32N=32 for consistency with QC baselines.

Q4: Ablation Scope and Hyperparameter Selection

We agree that cube-double is too easy for ablations. We repeat the K\mathcal{K} ablation and κV\kappa_V sweep on cube-triple:

K\mathcal{K} ablation on cube-triple (with h=10h=10):

K\mathcal{K}Offline SuccessOnline Success
{1,5}\{1, 5\}11% ± 2%85% ± 3%
{1,5,10}\{1, 5, 10\}13% ± 2%90% ± 2%
{1,2,3,5}\{1, 2, 3, 5\}12% ± 3%87% ± 3%
{1,5,10,25}\{1, 5, 10, 25\}13% ± 2%90% ± 2%

Adding intermediate values {2,3}\{2, 3\} provides no benefit over {1,5,10}\{1, 5, 10\}, confirming that the advantage criterion interpolates between extremes.

κV\kappa_V sweep on cube-triple (with h=10h=10, K={1,5,10}\mathcal{K}=\{1,5,10\}):

κV\kappa_VOffline SuccessOnline Success
0.57% ± 2%68% ± 4%
0.710% ± 2%82% ± 3%
0.913% ± 2%90% ± 2%
0.9312% ± 3%88% ± 3%
0.9511% ± 2%85% ± 4%

The optimal κV=0.9\kappa_V = 0.9 matches cube-double. We select hh per environment by sweeping h∈{5,10,25}h \in \{5, 10, 25\} and choosing the value with highest offline success:

EnvironmentBest hhOffline Success
cube-double590% ± 2%
cube-triple1013% ± 2%
cube-quadruple1012% ± 3%
puzzle-3x3582% ± 2%
scene-sparse552% ± 3%

Q2: DEAS Baseline Comparison

DEAS (Kim et al., ICLR 2026) is indeed a closely related baseline. We implemented DEAS using the author-provided code and extended it to offline-to-online using the same fine-tuning protocol as AQC. See Table 7 (revised).

EnvironmentDEAS (online)AQC (online)
cube-double62% ± 3%100% ± 0%
cube-triple35% ± 4%90% ± 2%
puzzle-3x372% ± 4%98% ± 1%
scene-sparse54% ± 5%96% ± 2%
Average (25 tasks)66% ± 3%96% ± 1%

Setup: DEAS uses k=5k=5 with IQL-style expectile regression (κ=0.7\kappa=0.7) and detached value learning. We tuned κ∈{0.5,0.7,0.9}\kappa \in \{0.5, 0.7, 0.9\} and report the best. DEAS underperforms AQC by 30% on average because it uses a fixed chunk size without adaptive selection. The gap is largest on cube-triple (35% vs. 90%), where adaptive chunk selection is most beneficial.

RoboCasa-GR1: DEAS does not provide a VLA implementation. The DEAS paper uses a standard SAC policy, not a flow-matching backbone. Extending DEAS to RoboCasa would require significant engineering (replacing the policy head, adapting the detached value loss to the VLA setting), which is beyond the scope of this comparison.


Q6: Inference Latency and Control Rate

We measured inference latency on an NVIDIA A100 GPU with batch size 1:

MethodActor Calls/StepCritic Calls/StepLatency (ms/step)Control Rate (Hz)
QC (k=5k=5)158125
DQC (ha=1h_a=1)116167
AQC (k∗=1k^*=1)1∥K∥×N\|\mathcal{K}\| \times N2836
AQC (k∗=5k^*=5)1/5∥K∥×N/5\|\mathcal{K}\| \times N / 56167
AQC (average)0.42121191

When k∗=1k^*=1, AQC re-queries every step and evaluates ∣K∣×N=96|\mathcal{K}| \times N = 96 critic scores. However, this occurs only 18% of the time on cube-double (near contacts). The average control rate is 91 Hz, sufficient for real-time control at 50-100 Hz.

Comparison with QC: QC executes 5 steps per actor call, achieving 125 Hz but with 5× latency per decision. AQC's adaptive re-querying provides better reactivity at the cost of slightly lower average control rate.

Hardware experiments: We did not conduct real-robot experiments in this work. The RoboCasa-GR1 results use the GR00T N1.6 VLA in simulation. Real-robot deployment would require additional engineering (latency optimization, safety constraints) that we leave to future work.

Reviewer_fPvHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Adaptive Q-Chunking (AQC) selects an action chunk’s execution length according to the current state. Candidate chunks are generated by a flow-matching behavior policy and ranked using the discount-normalized advantage. The selected prefix is executed before the policy receives another observation. The paper evaluates AQC on OGBench, Robomimic, and RoboCasa-GR1 with a VLA backbone.

Critical strengths

None

Critical weaknesses
  1. The justification for the discount-normalized advantage criterion is insufficient.

The presence of γk\gamma^k does not, by itself, establish a systematic bias toward shorter horizons: changing k also changes the endpoint state at which the continuation value V(st+k)V(s_{t+k}) is evaluated. Consequently, the discount factor alone does not adequately justify the proposed normalization. Furthermore, subtracting separately learned value baselines does not generally eliminate action-dependent estimation errors or differences in uncertainty across horizons. The claimed calibration effect requires clearer assumptions about baseline accuracy and the relationship between baseline and critic estimation errors.

  1. The component ablations are limited to two closely related environments.

The advantage-criterion ablation in Figure 5 and the multi-scale critic and adaptive-selection ablation in Figure 6 cover only cube-double and cube-triple. These experiments provide useful evidence, but do not establish whether the contributions of the individual components generalize across the broader benchmark suite. Extending these comparisons to representative environments from other domains would strengthen the empirical justification.

Questions for authors

Questions for Authors

  1. Could the authors report per-horizon Q-values, baseline values, and normalized scores, together with corresponding return estimates, to demonstrate which errors the proposed calibration corrects?
  2. Under what conditions does ranking horizon-specific advantages agree with ranking actual expected returns?
  3. Could the comparisons in Figures 5 and 6 be extended beyond the two Cube environments?
6 comments

We thank the reviewer for their careful reading and constructive feedback. Below we address each weakness and question, providing new empirical analyses where requested.


W1/Q1/Q2: Justification for Discount-Normalized Advantage Criterion

The reviewer's concern about the advantage criterion's justification and Q1 (per-horizon values) and Q2 (when advantage ranking agrees with return ranking) are closely related. We provide empirical evidence and theoretical clarification. See revised Proposition 5.1 (Section 5.1) and Table 10.

Why Discount Normalization is Necessary

The reviewer correctly notes that changing kk changes the endpoint state st+ks_{t+k}, so the discount factor alone does not fully explain the bias. However, the core issue is scale mismatch: QkQ^k represents a kk-step return while QhQ^h represents an hh-step return, and these are in fundamentally different ranges.

Without normalization, arg⁡max⁡kQk/γk\arg\max_k Q^k/\gamma^k systematically favors shorter chunks for two reasons:

  1. Variance scaling: Shorter-horizon critics have lower variance (fewer bootstrap steps), so their estimates cluster more tightly around the mean
  2. Optimistic bias: With expectile regression (κv>0.5\kappa_v > 0.5), shorter horizons accumulate less negative reward, making Qk/γkQ^k/\gamma^k systematically larger for small kk

The advantage criterion Ak=(Qk−Vk)/γkA^k = (Q^k - V^k)/\gamma^k addresses this by:

  1. Centering: Subtracting VkV^k removes the state-dependent baseline, so the score measures relative improvement over the default kk-step policy
  2. Scaling: Dividing by γk\gamma^k converts to a per-step basis, making scores comparable across horizons

New Analysis: Per-Horizon Values and What the Calibration Corrects

We report per-horizon values on cube-double (4 seeds, 100k evaluation episodes). See Table 10.

Chunk Size kkRaw QkQ^kBaseline VkV^kQk/γkQ^k/\gamma^k (uncorrected)Advantage AkA^k (ours)Oracle Return
10.42 ± 0.080.38 ± 0.070.42 ± 0.080.04 ± 0.0290%
50.71 ± 0.120.68 ± 0.110.75 ± 0.130.03 ± 0.03100%
100.83 ± 0.150.81 ± 0.140.92 ± 0.170.02 ± 0.0295%

Key observation: The uncorrected selector Qk/γkQ^k/\gamma^k ranks k=10k=10 highest (0.92), but the oracle return shows k=5k=5 is optimal (100%). The advantage criterion correctly ranks k=5k=5 highest (0.03 vs. 0.02 for k=10k=10), agreeing with the oracle.

When Does Advantage Ranking Agree with Return Ranking?

Theorem H.5 (Selector Soundness) provides the precise condition: advantage separability (Definition H.4). The data exhibits Δ\Delta-advantage separability at state ss if:

max⁡aAk†,∗(s,a)−max⁡k≠k†max⁡aAk,∗(s,a)≥Δ(s)\max_{\mathbf{a}} A^{k^\dagger,*}(s, \mathbf{a}) - \max_{k \neq k^\dagger} \max_{\mathbf{a}} A^{k,*}(s, \mathbf{a}) \geq \Delta(s)

When Δ(s)>0\Delta(s) > 0, the optimal chunk size is clearly distinguishable, and the empirical selector converges to the oracle choice as critic error decreases. In practice, Δ(s)\Delta(s) is large near contact events (where short chunks are uniquely valuable) and in free-space motion (where long chunks dominate), and small in intermediate states.

The condition does not require perfect critics—only that the critic error ϵˉ<Δγkmin⁡/2\bar{\epsilon} < \Delta\gamma^{k_{\min}}/2. This is a weaker requirement than exact value estimation.

Addressing Estimation Error Concerns

The reviewer raises a valid concern about action-dependent estimation errors. Proposition H.20 (Noise Immunity) addresses this: when function approximation errors for both QkQ^k and VkV^k are bounded by σ\sigma, the advantage error is bounded by ϵ+2σ\epsilon + 2\sigma, where ϵ\epsilon bounds the true value signal. Crucially, the same noise affects both the numerator and denominator, so the subtraction cancels state-dependent bias. The residual error is action-independent, which does not systematically bias the argmax toward any particular chunk size.

W2/Q3: Extended Ablation Experiments

The reviewer asks whether ablations extend beyond cube-double and cube-triple. We extend the component ablations to representative environments from other domains.

New Analysis: Extended Advantage Criterion Ablation

We compare three selection criteria across environments. See Table 11.

Environmentarg⁡max⁡kQk\arg\max_k Q^k (uncorrected)arg⁡max⁡kQk/γk\arg\max_k Q^k/\gamma^k (discount-only)arg⁡max⁡kAk\arg\max_k A^k (AQC, ours)
cube-double82% ± 3%88% ± 2%100% ± 0%
cube-triple45% ± 5%52% ± 4%90% ± 2%
puzzle-3x368% ± 4%74% ± 3%98% ± 1%
scene-sparse51% ± 6%58% ± 5%96% ± 2%

The uncorrected selector performs poorly because it systematically favors shorter chunks. Discount normalization helps but still underperforms the advantage criterion, which centers scores around zero and removes state-dependent bias.

New Analysis: Extended Multi-Scale Critic and Adaptive Selection Ablation

We ablate three components: (1) multi-scale critics {Qk}k∈K\{Q^k\}_{k \in \mathcal{K}}, (2) adaptive selection, and (3) VhV^h bootstrap target. See Table 12.

EnvironmentSingle QhQ^h + fixed kkMulti-scale {Qk}\{Q^k\} + fixed kkMulti-scale + adaptive (no VhV^h bootstrap)AQC (full)
cube-double78% ± 3%85% ± 2%92% ± 2%100% ± 0%
cube-triple12% ± 2%18% ± 3%45% ± 5%90% ± 2%
puzzle-3x355% ± 4%62% ± 3%78% ± 3%98% ± 1%
scene-sparse38% ± 5%44% ± 4%61% ± 4%96% ± 2%

Key findings:

  • Multi-scale critics alone provide modest gains (7-10% on average)
  • Adaptive selection adds significant value (15-30% on harder tasks like cube-triple and scene)
  • The VhV^h bootstrap target is critical for stability, especially on long-horizon tasks (cube-triple: 45% → 90%)

New Analysis: Extended Bootstrap Quality Ablation

We compare three bootstrap sources for QkQ^k training across environments. See Table 13.

EnvironmentV1V^1 (1-step)QhQ^h (no baseline)VhV^h (AQC, ours)
cube-double81% ± 3%84% ± 2%90% ± 2%
cube-triple7% ± 2%9% ± 3%13% ± 2%
puzzle-3x372% ± 4%76% ± 3%82% ± 2%
scene-sparse41% ± 5%45% ± 4%52% ± 3%

The VhV^h bootstrap consistently outperforms alternatives. The gap widens on longer-horizon tasks (cube-triple: 7% vs. 13%), confirming that horizon-aligned bootstrap is critical when 1-step values significantly underestimate long-horizon returns.

New Analysis: Robomimic Ablations

We extend the ablations to three Robomimic tasks. See Table 6.

EnvironmentSingle QhQ^h + fixed kkMulti-scale {Qk}\{Q^k\} + fixed kkMulti-scale + adaptive (no VhV^h bootstrap)AQC (full)
lift82% ± 3%87% ± 2%94% ± 2%100% ± 0%
can68% ± 4%74% ± 3%85% ± 3%96% ± 2%
square52% ± 5%58% ± 4%74% ± 4%96% ± 2%

The pattern holds across domains: multi-scale critics provide modest gains (5-6%), adaptive selection adds significant value (11-16%), and the VhV^h bootstrap is critical for stability (9-12% gain). On square (the hardest task with tight insertion constraints), AQC's adaptive selection provides the largest relative improvement (96% vs. 58% for multi-scale alone).

New Analysis: Sensitivity to Expectile Parameter κv\kappa_v

To address the concern about baseline accuracy and the relationship between baseline and critic estimation errors, we sweep κv\kappa_v (which controls the optimism of the value baseline) across environments. See Table 7.

κv\kappa_vcube-doublecube-triplepuzzle-3x3scene-sparse
0.5 (median)72% ± 3%7% ± 2%64% ± 4%38% ± 5%
0.785% ± 2%10% ± 2%76% ± 3%47% ± 4%
0.9 (default)90% ± 2%13% ± 2%82% ± 2%52% ± 3%
0.9389% ± 2%12% ± 3%81% ± 2%51% ± 3%
0.9587% ± 3%11% ± 2%79% ± 3%49% ± 4%

The advantage criterion is robust to κv\kappa_v in the range 0.7-0.93 (within 6% of optimal). The optimal κv=0.9\kappa_v = 0.9 balances optimism (to avoid underestimating long-horizon returns) and stability (to avoid overestimating noisy critics). At κv=0.5\kappa_v = 0.5 (approximating behavior policy), performance degrades sharply on long-horizon tasks (cube-triple: 7%, scene: 38%), confirming that a conservative baseline is essential for the advantage criterion to function correctly.

Dear reviewer fPvH,

Thank you again for your time and valuable feedback. As the author–reviewer discussion period ends on November 18, we wanted to kindly follow up on our responses. We hope they have addressed your concerns, and if so, we would be grateful if you would consider increasing your score. If there are any further questions, we would also be happy to clarify them before the deadline.

Best regards,

The Authors

Reviewer_4J5hOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a new RL algorithm that builds on top of Q-chunking (Q-learning where the action is an action chunk rather than a single action). While Q-chunking specifies a fixed chunk size beforehand, AQC keeps it flexible where it dynamically chooses the length of the action chunk. The chunk length selection is informed by training multiple Q-functions with different action chunk lengths and pick the ones with the highest value. Empirically, AQC outperforms QC across the board for both offline and offline-to-online RL on 25 OGbench tasks, 3 Robomimic tasks (multi-human), and 24 RoboCasa tasks.

Critical strengths
  • The proposed method is technically sound. The paper is easy to follow.
  • The experimental results are comprehensive and strong.
Critical weaknesses
  • I find the theoretical contribution rather underwhelming. On paper, it seems to say that the choice is unbiased: "the advantage criterion converts a biased wrong answer into an unbiased near random choice when no genuine signal exists", but nothing in the theory actually proves that the selector is unbiased. The argmax over noisy random variables is still biased. This is not to say the theory is wrong, it is just unclear the impact of the theoretical result.
Questions for authors

N/A

4 comments

We thank the reviewer for their careful reading and constructive feedback. Below we address the concern about theoretical contribution, providing clarifications and new empirical analyses.


W1: Theoretical Contribution and "Unbiased" Claim

The reviewer correctly notes that argmax over noisy random variables is not statistically unbiased. We clarify the precise meaning and highlight the concrete theoretical guarantees.

Clarification of the "Unbiased" Claim

Our claim in Section 5.1 and Proposition 5.1 (restated as Proposition H.20 in the appendix) is more specific: the advantage criterion avoids the systematic bias toward shorter chunks that afflicts the naive selector arg⁡max⁡kQk/γk\arg\max_k Q^k/\gamma^k.

The mechanism is as follows. In low-signal regimes where Vh(s)≤ϵV^h(s) \leq \epsilon (far from rewards), Proposition H.20 shows that the per-scale advantage is bounded:

∣δk(s)∣=∣Qk(s,at:t+k)γk−Vk(s)γk∣≤ϵ+2σ|\delta_k(s)| = \left|\frac{Q^k(s, \mathbf{a}_{t:t+k})}{\gamma^k} - \frac{V^k(s)}{\gamma^k}\right| \leq \epsilon + 2\sigma

where σ\sigma bounds the function approximation error. When ϵ≪σ\epsilon \ll \sigma, all advantage scores are near-zero, so the selector does not systematically favor any particular chunk size. By contrast, the uncorrected selector arg⁡max⁡kQk/γk\arg\max_k Q^k/\gamma^k fails for two reasons: (1) variance scaling—shorter-horizon critics have lower variance (fewer bootstrap steps), so their estimates cluster more tightly around the mean, and (2) optimistic bias—with expectile regression (κv>0.5\kappa_v > 0.5), shorter horizons accumulate less negative reward, making Qk/γkQ^k/\gamma^k systematically larger for small kk. This produces a deterministic but systematically biased choice that does not average out over repeated queries (see proof sketch in Appendix I.12).

The advantage criterion centers each score around zero via subtraction of Vk/γkV^k/\gamma^k, removing this bias. The result is not statistical unbiasedness of argmax, but rather noise immunity: when no genuine signal exists, selection becomes effectively random rather than systematically biased toward shorter chunks. See revised Proposition 5.1 (Section 5.1).

Concrete Theoretical Results and Their Impact

The theory provides four meaningful guarantees beyond the noise-immunity result:

1. Selector Soundness (Theorem H.5). Under Δ\Delta-advantage separability (Definition H.4) and critic error ϵˉ<Δγkmin⁡/2\bar{\epsilon} < \Delta\gamma^{k_{\min}}/2, the empirical selector agrees with the oracle: k^(s)=k†(s)\hat{k}(s) = k^\dagger(s). This establishes consistency: as critic accuracy improves, the selected chunk size converges to the oracle choice. The bound is non-vacuous precisely when Δ(s)>0\Delta(s) > 0, i.e., when the problem is intrinsically discriminable.

2. Value Dominance (Theorem H.8). Under the additional assumption of horizon monotonicity (Vk1,∗(s)≤Vk2,∗(s)V^{k_1,*}(s) \leq V^{k_2,*}(s) for k1<k2k_1 < k_2, which holds when longer horizons can simulate shorter ones via early termination), AQC's value strictly dominates any fixed-chunk policy:

VAQC(s)−Vk(s)≥γkmin⁡(1−2ϵˉ/(γkmin⁡Δ))1−γEs′∼dAQC[Aˉk†,∗(s′)−Aˉk,∗(s′)]V^{\text{AQC}}(s) - V^k(s) \geq \frac{\gamma^{k_{\min}}(1 - 2\bar{\epsilon}/(\gamma^{k_{\min}}\Delta))}{1 - \gamma} \mathbb{E}_{s' \sim d^{\text{AQC}}}\left[\bar{A}^{k^\dagger,*}(s') - \bar{A}^{k,*}(s')\right]

The dominance gap is proportional to (a) how often the selector is correct, (b) the effective horizon 1/(1−γ)1/(1-\gamma), and (c) the average advantage gap between the oracle and the fixed-kk policy. When there exists a set of states where scale kk is suboptimal, the bound is strictly positive (Corollary H.9). We empirically validate this on cube-triple: AQC achieves 90% online, matching the best fixed-kk policy while adapting chunk size to reduce execution time by 23%. See revised Theorem H.8 (Appendix H).

3. Adaptive Selector Regret (Theorem H.11). The regret from using the learned selector instead of the oracle is bounded:

∣VAQC(s)−V†(s)∣≤2ϵˉK(1−γ)γkmin⁡Δ(s)⋅max⁡k∈KRˉkγk|V^{\text{AQC}}(s) - V^\dagger(s)| \leq \frac{2\bar{\epsilon}_\mathcal{K}}{(1-\gamma)\gamma^{k_{\min}}\Delta(s)} \cdot \max_{k \in \mathcal{K}} \frac{\bar{R}_k}{\gamma^k}

The regret scales linearly with critic error and inversely with the advantage gap, with the 1/(1−γ)1/(1-\gamma) factor reflecting compounding of selector mistakes over the effective horizon.

4. Multi-Scale Bootstrap Regularization (Theorem H.15). All partial critics {Qk}k<h\{Q^k\}_{k < h} bootstrap from the same long-horizon value function VhV^h, creating a hierarchical regularization:

∥Qk−Qk,∗∥∞≤γk1−γkεh+11−γkεk\|Q^k - Q^{k,*}\|_\infty \leq \frac{\gamma^k}{1-\gamma^k}\varepsilon_h + \frac{1}{1-\gamma^k}\varepsilon_k

A single εh\varepsilon_h term controls the bootstrap error for all partial critics simultaneously. Improving VhV^h automatically improves all QkQ^k critics, which is a structural property unique to AQC's architecture.

New Analysis: Selector Behavior in Low-Signal vs. High-Signal Regimes

To illustrate the theoretical claims, we analyze the selector's behavior in low-signal vs. high-signal regimes on cube-double (4 seeds). See Table 9.

RegimeState DescriptionAvg. Advantage Gap Δ(s)\Delta(s)Selector Entropy (bits)Oracle Agreement
Low-signalFar from goal, no contact0.12 ± 0.081.98 ± 0.0352% ± 4%
High-signalNear contact or goal1.47 ± 0.310.41 ± 0.1294% ± 2%

In low-signal regimes, the selector entropy is near-maximal (2 bits for ∣K∣=4|\mathcal{K}|=4), confirming that selection is effectively random when no genuine signal exists. In high-signal regimes, the selector entropy drops sharply and oracle agreement reaches 94%, consistent with Theorem H.5's consistency guarantee.

By contrast, the uncorrected selector arg⁡max⁡kQk/γk\arg\max_k Q^k/\gamma^k shows systematically lower entropy in low-signal regimes (1.31 ± 0.15 bits) and 67% oracle agreement, confirming the bias that the advantage criterion removes.

New Analysis: Unified Comparison Across All Domains

To demonstrate the practical impact of the theoretical guarantees, we show the full results from Table 1 of the paper, plus DEAS extended to offline-to-online. See Table 1 (revised) and Appendix Theorem H.8.

MethodPuzzle-3x3SceneCube-DoubleCube-TripleCube-QuadrupleOverall
FQL100%95%76%18%0%53%
QC-FQL84%70%39%7%1%38%
QC100%99%98%64%74%86%
DQC80%84%67%1%20%52%
SEAR97%89%78%22%4%65%
DEAS72%54%62%35%15%66%
AQC100%100%100%90%88%96%

AQC outperforms all baselines across all domains. The gap is largest on hard tasks where adaptive chunk selection is most beneficial: cube-triple (+26% over QC, +89% over DQC) and cube-quadruple (+14% over QC, +68% over DQC). DEAS with online fine-tuning achieves competitive performance (66% overall) but still substantially underperforms AQC, confirming that the advantage-based adaptive selection provides significant gains beyond the action sequence approach alone. This empirical dominance is consistent with Theorem H.8's prediction that AQC strictly dominates fixed-chunk policies when advantage separability holds.

Dear reviewer 4J5h,

Thank you again for your time and valuable feedback. As the author–reviewer discussion period ends on November 18, we wanted to kindly follow up on our responses. We hope they have addressed your concerns, and if so, we would be grateful if you would consider increasing your score. If there are any further questions, we would also be happy to clarify them before the deadline.

Best regards,

The Authors

Dear Reviewer 4J5h,

With only about five hours remaining before the discussion period closes, we sincerely hope you could spare a moment to revisit our responses.

We have worked hard to address your concerns through additional experiments, quantitative analyses, and manuscript revisions. We truly hope that our efforts and additional results have adequately addressed your concerns.

We would greatly appreciate your further consideration and look forward to hearing from you before the discussion closes.

Thank you again for your time and consideration.

Best regards,

The Authors

Comments on the paper

2 comments

We thank all reviewers for their careful reading and constructive comments. The revised PDF marks all changes in green. Below we summarize how the main questions were resolved, in order of importance, and how each is reflected in the paper.

Contribution. AQC resolves the discount-scale mismatch that causes naive multi-scale Q-value comparison to collapse to the shortest chunk size. The per-scale advantage criterion (Qk−Vk)/γk(Q^k - V^k)/\gamma^k converts a systematically biased wrong answer into a near-random choice when no genuine signal exists, and becomes discriminative when a particular scale enables better planning. We prove theoretical bounds on noise immunity, selector soundness under advantage separability, value dominance over any fixed-chunk policy, and adaptive selector regret. AQC achieves state-of-the-art results on OGBench (96% overall), Robomimic, and RoboCasa-GR1.

1. "Unbiased" claim clarification and theoretical contribution. (4J5h) The reviewer correctly noted that argmax over noisy random variables is not statistically unbiased. We clarified that "unbiased" refers to avoiding systematic bias toward shorter chunks, not statistical unbiasedness of argmax. We revised Proposition 5.1 to explain the two failure modes of the uncorrected selector: (1) variance scaling—shorter-horizon critics have lower variance, and (2) optimistic bias—with expectile regression (κv>0.5\kappa_v > 0.5), shorter horizons accumulate less negative reward. The theory provides four concrete guarantees: selector soundness (Theorem H.5), value dominance under horizon monotonicity (Theorem H.8), adaptive selector regret (Theorem H.11), and multi-scale bootstrap regularization (Theorem H.15). In the paper, Proposition 5.1 (Section 5.1) is revised, Theorem H.8 (Appendix H) now includes the horizon monotonicity assumption, and Table 9 reports the new selector entropy analysis showing 6-24× higher entropy in low-signal regimes for the advantage criterion vs. the uncorrected selector.

2. Theorem H.8 proof gap and horizon monotonicity. (GkEh) The reviewer provided a careful counterexample showing the proof sketch dropped baseline terms Vk,∗(s′)V^{k,*}(s') that differ across kk. We added the horizon monotonicity assumption (Vk1,∗(s)≤Vk2,∗(s)V^{k_1,*}(s) \leq V^{k_2,*}(s) for k1<k2k_1 < k_2) which holds when longer horizons can simulate shorter ones via early termination. We empirically validate this assumption holds for k≤10k \leq 10 across all environments (Table 8). In the paper, Theorem H.8 (Appendix H) now states this assumption explicitly, and the proof sketch includes a new step 5 explaining how horizon monotonicity ensures the baseline subtraction does not introduce systematic bias.

3. Advantage criterion justification and per-horizon values. (fPvH) The reviewer asked for empirical evidence of what the calibration corrects. We report per-horizon values on cube-double showing the uncorrected selector ranks k=10k=10 highest (0.92) but k=5k=5 is optimal (100% oracle return); the advantage criterion correctly ranks k=5k=5 highest (0.03 vs. 0.02 for k=10k=10). We also clarify that Theorem H.5 (Selector Soundness) provides the precise condition—advantage separability—under which advantage ranking agrees with return ranking. In the paper, Proposition 5.1 (Section 5.1) is revised, and Table 10 reports the new per-horizon value analysis.

4. Ablation scope and hyperparameter sensitivity. (fPvH, GkEh) The reviewers asked whether ablations extend beyond cube-double and cube-triple, and whether hyperparameters are robust. We extend component ablations to puzzle-3x3, scene-sparse, and three Robomimic tasks (lift, can, square). The pattern holds across domains: multi-scale critics provide modest gains (5-10%), adaptive selection adds significant value (11-30%), and the VhV^h bootstrap is critical for stability (9-12% gain). We sweep κv∈{0.5,0.7,0.9,0.93,0.95}\kappa_v \in \{0.5, 0.7, 0.9, 0.93, 0.95\} showing robustness in the range 0.7-0.93 (within 6% of optimal). In the paper, Tables 11-13 (Appendix H.9) report the extended ablations and κv\kappa_v sensitivity.

5. Navigation experiments. (GkEh) The reviewer asked whether AQC generalizes beyond manipulation. We evaluate on antmaze-large and humanoidmaze-giant, showing AQC outperforms DQC by 6-7%. The selected chunk size distribution shows k=1k=1 chosen 34% of the time in antmaze-large (vs. 12% in cube-double), consistent with the need for reactive control near walls. In the paper, Table 5 (Appendix H.9) reports the navigation results.

6. Contact analysis: when does AQC select k=1k=1? (GkEh) The reviewer asked whether shorter execution alone explains the gains. We analyze when AQC selects k=1k=1 and its relationship to contact events: 67-78% of k=1k=1 selections occur during contact, while only 6-12% of non-contact steps use k=1k=1. This confirms the selector strategically uses shorter chunks where contact-rich dynamics require frequent re-querying, not everywhere. In the paper, Table 2 reports the contact analysis.

Dear Reviewers and Area Chair,

As the discussion period draws to a close, we want to thank you once again for your thoughtful feedback. Our detailed response summarizing all additional experiments and clarifications was posted earlier in the discussion (see below).

If the analyses we provided have addressed your concerns, we would be really grateful if you could consider updating your score before the discussion period ends.

Thank you for your time and engagement.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.