acceptodds
Under review as a conference paper at ICLR 2027

Value-Guided Adaptive Action Chunking for Reinforcement Learning

Abstract

Action chunking emerged as a pivotal technique in imitation learning, enabling policies to predict cohesive action sequences rather than single actions. Recently, this approach has expanded to reinforcement learning (RL), enhancing behavioral consistency and reducing bootstrapping errors in value function estimation. However, existing methods rely on a fixed chunk length, creating a performance bottleneck as the optimal length varies across states and tasks. In this paper, we propose **A**daptive Action **CH**unking (ACH), a novel offline-to-online RL algorithm that dynamically modulates chunk length during both training and inference. For each state, ACH takes the action-value of each candidate chunk length as its optimality criterion. We estimate the action-values of all candidate chunk lengths simultaneously in a single forward pass by employing a causal Transformer-based value function. Based on these values, the agent adaptively selects the most effective chunk length for the current state. Evaluated on 34 challenging tasks, ACH outperforms fixed-length baselines in most environments, demonstrating enhanced learning efficiency. Moreover, ACH improves the performance of vision-language-action (VLA) models, demonstrating its applicability to large-scale models.

Primary area:
reinforcement learning
Keywords:
adaptive action chunking, offline-to-online reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PNwfOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes ACH, an offline-to-online RL algorithm that adaptively changes the action chunking size. It learns a causal Transformer-based value function, and adaptively selects the action chunk size from the Q values. ACH consistently outperforms prior offline-to-online works, including OGBench and SimplerEnv (with VLAs).

Critical strengths

S1. Method is straightforward and simple.

  • The method trains a Transformer value function and extracts the policy with DDPG + BC (from flow BC policy).
  • Action chunk size is adaptively selected via Boltzman soft policy.
  • This can be seamlessly applied to pre-trained policies such as VLAs.

S2. Strong experimental results.

  • ACH consistently outperforms prior offline-to-online methods in OGBench and SimplerEnv with VLAs.
  • Also, ablation studies and qualitative anlaysis (Section 4.2) support that adaptive action chunking is important and ACH's objective improves the policy via adaptive action chunking.
Critical weaknesses

W1. Lack of theoretical understanding of the objectives

  • The policy is decomposed to two parts, action chunking policy πθ\pi_{\theta} and size policy πL\pi_L, where πL\pi_L is a boltzman policy.
  • While individual design choices are reasonable, the paper does not contain where the policy and Q function will converge from these objectives.
  • For example, if there is no BC constraint, the algorithm will converge to a markovian policy without action chunking, as markovian policy is always as good as nonmarkovian policy due to reactivity.
  • It will be great if the authors can provide what policy / Q function is going to be learned by this methodology.
Questions for authors

Q1. Theoretical understanding of the objective

  • Would it be possible to provide explanation or intuition on what policy / Q function is going to be learned by this methodology?
  • This will help the readers on understanding the objective of the algorithm.
6 comments
AuthorsOpenReview ↗

Authors' response to reviewer PNwf

We thank the reviewer for the thoughtful review. The question of what the objectives of ACH learn motivated a new theoretical analysis, which we added as Appendix B.1 and summarized in Section 3.4 of the revised manuscript (changes are marked in blue). We summarize the results below.

R3-1. On the theoretical understanding of the objectives (W1, Q1)

The overall procedure of ACH is captured by the composite policy μ=(πθ,πL)\mu=(\pi_\theta,\pi_L), which at each decision state samples an action chunk from πθ\pi_\theta and an execution length from πL\pi_L, executes the corresponding prefix open-loop, and makes its next decision at the reached state.

Since an action chunk is a semi-Markov option (Sutton et al., 1999), learning with adaptive chunks forms a Semi-MDP (SMDP), and standard SMDP theory can be adapted to our setting. Assuming exact policy evaluation and improvement in the tabular setting with deterministic transitions, we show the following:

What the Q function learns (Proposition 1, policy evaluation). For a fixed μ\mu, the value functions satisfy

Qμ(st,at:t+n)=Rtn+γn+1Vμ(st+n+1),Vμ(st)=Eat:t+h′∼πθ, ℓ∼πL[Qμ(st,at:t+ℓ′)],Q^\mu(s_t,\boldsymbol{a}_{t:t+n}) = R_t^n+\gamma^{n+1}V^\mu(s_{t+n+1}),\qquad V^\mu(s_t)=\mathbb{E}_{\boldsymbol{a}'_{t:t+h}\sim\pi_\theta,\ \ell\sim\pi_L}\big[Q^\mu(s_t,\boldsymbol{a}'_{t:t+\ell})\big],

which are the SMDP Bellman expectation equations of μ\mu and have a unique solution (Qμ,Vμ)(Q^\mu,V^\mu). The two targets of the mean-square loss in Eq. (7) correspond to these two equations. In other words, the critic of ACH learns the value of every prefix under the current chunk policy and the current length policy.

How the policy is improved (Proposition 2, policy improvement). If a new composite policy μ′\mu' satisfies Eμ′[Qμ(st,at:t+ℓ′)]≥Vμ(st)\mathbb{E}_{\mu'}[Q^\mu(s_t,\boldsymbol{a}'_{t:t+\ell})]\ge V^\mu(s_t) at every state, then Vμ′≥VμV^{\mu'}\ge V^\mu. The improvement condition holds (i) when the chunk policy is kept and the length is selected greedily, which is the selection used at evaluation, and (ii) when the chunk policy additionally generates a chunk that maximizes the best prefix value. The value term in Eq. (8) similarly encourages the actor to generate chunks with higher expected prefix values, while Proposition 2 specifies the condition under which an update guarantees improvement.

Where the values converge (Theorem 1, policy iteration). Alternating exact evaluation of Proposition 1 and globally greedy improvement of Proposition 2 (ii) gives Vμ0≤Vμ1≤⋯≤V∗V^{\mu_0}\le V^{\mu_1}\le\cdots\le V^* and Vμk→V∗V^{\mu_k}\to V^*, where V∗V^* is the optimal value function of the MDP. Thus, the value function converges to the optimal value function of the original MDP under this exact policy iteration.

Optimal chunks (Theorem 2). There exists an optimal composite policy μ∗\mu^* whose chunk attains V∗V^* at every prefix length. Hence, restricting the policy to open-loop chunks with adaptive lengths does not lose optimality.

Chunking and the optimal Markovian policy. A policy with action chunking cannot be better than the optimal Markovian policy, which selects every action from the current state. Under the exact policy iteration of Theorem 1, the values converge to the value of the optimal Markovian policy. By Theorem 2, attaining the optimal value does not require a chunk length of one. However, the chunk length matters for stochastic and suboptimal policies encountered during learning, and in particular for the behavior-regularized policy of Eq. (8). For any such μ\mu, the values of adjacent lengths satisfy

Qμ(st,at:t+n)−Qμ(st,at:t+n−1)=γn(Qμ(st+n,at+n)−Vμ(st+n)).Q^\mu(s_t,\boldsymbol{a}_{t:t+n})-Q^\mu(s_t,\boldsymbol{a}_{t:t+n-1})=\gamma^n\big(Q^\mu(s_{t+n},a_{t+n})-V^\mu(s_{t+n})\big).

Extending a chunk by one step increases the value exactly when the pre-planned action at+na_{t+n} is better than replanning at st+ns_{t+n}, i.e., drawing a new chunk and a new length. When the reached state is consistent with the plan, replanning can introduce the sampling noise of the policy without improving the action, and longer chunks are favored. When the pre-planned action, which is generated from sts_t alone, no longer suits the reached state, shorter chunks are favored. Comparing the values of all prefixes lets ACH balance these two cases at each state, which is consistent with its improved sample efficiency during learning.

The practical algorithm. In Eq. (8), the BC term keeps the actor close to the behavior policy, and during online interaction the soft πL\pi_L provides exploration over chunk lengths. Theorem 1 assumes exact evaluation and globally greedy improvement, whereas the practical algorithm uses function approximation and gradient updates of the behavior-regularized actor. Removing the BC term alone does not make these updates equivalent to the policy iteration of Theorem 1. We state the scope of the analysis at the end of Appendix B.1.


We thank the reviewer again for the constructive suggestion and would be happy to provide any further clarification.

Thank you for the clarification and providing the theoretical analysis.

However, I think the proposed theoretical result does not address my concern on "what policy / Q function is going to be learned by this methodology", as 1) this result does not incorporate BC constraint, 2) therefore this result again shows that it will converge to a markovian optimal policy (or action chunked policy that is equivalent to markovian policy).

I'll maintain the positive score.

Thank you for the quick response. We appeciate your positive evaluation.

As noted, when the behavior regularization term is not present in Eq. (8), then Eqs. (7) and (8) form a case of continuous version of policy iteration for semi-MDP. However, with the behavior regularization term, analysis is difficult. Especially, the regularization term is in a soft form which makes analysis more difficult. We conjecture that if we change the soft behavior constraint to a hard constraint as min⁡θL(θ)=Eat:t+h′∼πθ[−Eℓ∼πLQϕ(st,at:t+ℓ′)]such thatEat:t+h′∼πθ[∣at:t+h′−β(st)∣2]≤κ,\min_{\theta} L(\theta) = \mathbb{E}_{a'_{t:t+h}\sim \pi_\theta} [-\mathbb{E}_{\ell \sim \pi_L} Q_\phi(s_t, a'_{t:t+\ell})] \quad \text{such that} \quad \mathbb{E}_{a'_{t:t+h}\sim \pi_\theta} \left[\\|a'_{t:t+h} - \beta(s_t)\\|^2\right] \le \kappa, then the constrained policy iteration may converge to a policy maximizing Q under the behavior regularization. It seems that this conjecture requires further in-depth analysis. If the result is available, we will get back to you again.

Thank you.

Authors

Reviewer_x887OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ACH, which builds on the one-step variant of Q-chunking and chooses, at each decision point, how many actions of a generated chunk to execute. A causal Transformer critic returns V(st)V(s_t) and all prefix values in one pass and is trained with nn-step targets, while VV and the actor use prefix values weighted by πL\pi_L, a softmax over them. Lengths are sampled from πL\pi_L online and chosen by arg-max at evaluation. ACH is best on four of six OGBench environments and on robomimic square and transport, and lifts SimplerEnv success on a frozen GR00T N1.7 from 59.4% to 71.0% (WidowX) and from 44.5% to 71.9% (Google Robot).

Critical strengths

1. A simple mechanism that fits existing chunked RL.

Comparing prefixes of one chunk isolates the effect of length, one forward pass gives every prefix value, and one length distribution drives the value target, the actor and execution.

2. Clear gains over the fixed-length backbone on long-horizon manipulation.

Relative to its QC backbone, ACH improves cube-quadruple from 33 to 52, puzzle-4x4 from 38 to 90 and robomimic transport from 22 to 39. All three are long-horizon, multi-stage tasks.

Critical weaknesses

1. Limited comparison with strong chunked baselines.

The QC baseline appears to be the one-step QC-FQL variant, whereas the main method of [1] is best-of-N QC, which [1, Table 1] reports at 64 on cube-triple compared with 53 for QC-FQL. Including best-of-N QC under the same data and budget, ideally also on the remaining OGBench tasks of [1] (cube-double, scene-sparse and puzzle-3x3-sparse), would make it clearer how much ACH adds over this strong baseline. Sec. 5 also notes that Transformer-critic chunking methods, including SEAR [2], suffer from limited sample efficiency due to a fixed chunk length. Since SEAR uses a similar prefix critic and already executes random prefixes during training, a direct comparison would help support this statement.

Regarding Table 4, when the fixed length is chosen per environment, its reported mean is close to that of ACH on antmaze-giant (H1: 74 vs. 72), antsoccer (H3: 82 vs. 84) and cube-triple (H10: 70 vs. 67), so the gain over a tuned fixed length appears mainly in the other three environments. In Table 1, ReBRAC on antmaze-giant (99 vs. 72) and FINO on humanoidmaze-medium (92 vs. 82) also perform better than ACH. In Fig. 12, QC reaches higher success earlier than ACH on tasks 1 and 2 of both cube environments, so it is unclear whether the sample-efficiency advantage holds in general.

2. The contribution of adaptive lengths is difficult to separate from the TOP-ERL-style critic.

As Sec. 3.2 notes, the causal Transformer critic with a VV output, the nn-step targets for all prefixes and the bootstrapping from VV follow TOP-ERL [3], and the new components are the πL\pi_L weighting in Eqs. (7)-(8) and the selection rule in Eq. (9). The fixed-length results (QC and H1 to H10 in Table 4) use the QC MLP critic, while the Random, Uniform and Greedy ablations keep the Transformer critic. These ablations show that value-based selection and weighting outperform random lengths and uniform weights with the same critic, which I appreciate, but they do not compare against a fixed execution length with ACH's critic. Since such critics can improve results on their own [3, 4], this control would help isolate the benefit of adaptive lengths.

Questions for authors
  1. Values and the SMDP claim. It would help to state in Sec. 3 that QQ and VV evaluate the policy with the soft πL\pi_L (Eq. (5) refers to a generic π\pi), while evaluation uses greedy selection. App. B also states that Eq. (7) avoids off-policy bias. With stochastic transitions and logged actions that react to intermediate states, however, the conditional replay target may exceed the open-loop value of a chunk and thus favor longer prefixes. Could the authors clarify under which open-loop consistency condition [5] Eq. (7) recovers Eq. (10), and whether the replay data satisfy it?

  2. Offline stage. Offline success is close to zero on most tasks (0 to 1% on all antsoccer tasks and 0 to 3% on all antmaze-giant and cube tasks). I wonder what pre-training contributes in these cases. An ACH run that starts online with the same offline data in the buffer but without pre-training, as in RLPD, would help clarify this.

  3. Maximum length and α\alpha. App. C.4 lowers the maximum length to 5 on humanoidmaze, since learning with 10 becomes difficult. Does the statement in App. D.3 that no per-environment tuning is needed refer only to the executed length? I also could not find an ablation of α\alpha. Because α\alpha constrains the whole chunk while the value term in Eq. (8) covers only the weighted prefixes, it may also affect the selected lengths. How sensitive are the performance and the length distribution to α\alpha, and how does ACH perform on humanoidmaze with a maximum length of 10?

  4. Growth of the chunk length. Could the authors clarify whether Fig. 10 reports lengths sampled during training or the arg-max lengths used at evaluation? App. D.5 attributes the increase to trajectory stitching, but shifts in the visited states could lead to a similar trend, and so could a growing spread of prefix values under a fixed β\beta if the lengths are sampled and longer prefixes tend to score higher. Comparing the preferred length at fixed states across checkpoints would help distinguish these explanations.

  5. Selection signal. Which β\beta is used on OGBench and robomimic, and are the prefix values normalized before Eq. (9)? It would also be helpful to see whether Monte-Carlo returns, measured from the same state with the same full candidate chunk and the continuation policy that the critic evaluates, support the critic's ranking of prefixes.

  6. Cost in the VLA setting. App. D.4 reports the cost only on antmaze-giant. How many backbone calls per environment step does ACH require in SimplerEnv, and what is its end-to-end decision latency compared with Base and AAC?

References:

[1] Li et al. "Reinforcement Learning with Action Chunking." NeurIPS 2025.

[2] Nagy et al. "SEAR: Sample Efficient Action Chunking Reinforcement Learning." NeurIPS 2026.

[3] Li et al. "TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learning." ICLR 2025.

[4] Tian et al. "Chunking the Critic: A Transformer-based Soft Actor-Critic with N-step Returns." ICLR 2026.

[5] Li et al. "Decoupled Q-Chunking." ICLR 2026.

9 comments
AuthorsOpenReview ↗

Authors' response to reviewer x887 (1/6)

We thank the reviewer for the careful reading and constructive questions, which led to several new experiments and a new theoretical analysis in the revised manuscript (changes are marked in blue). We address each point below.

R2-1. Comparison with best-of-N QC and the remaining OGBench tasks of QC (W1)

As the reviewer points out, our QC baseline is the QC-FQL variant, on which ACH is also built. We additionally ran the best-of-NN variant of QC with N=32N=32 under the same data and budget on all six environments of Table 1 with 4 seeds.

QC-FQLQC (best-of-N)ACH
antmaze-giant0±0 → 47±80±0 → 72±111±1 → 72±2
antsoccer-arena10±2 → 67±230±10 → 80±50±0 → 84±1
cube-quadruple-10M1±1 → 33±38±4 → 42±90±0 → 52±6
cube-triple4±2 → 57±46±2 → 61±51±1 → 67±3
humanoidmaze-medium15±2 → 32±439±3 → 71±745±2 → 82±1
puzzle-4x411±2 → 38±418±3 → 52±913±3 → 90±8

The best-of-NN variant improves over QC-FQL in all environments, in line with [1]. ACH performs comparably to it on antmaze-giant and antsoccer-arena and outperforms it on the remaining four environments, with the largest gap on puzzle-4x4. Additionally, ACH achieves this with a single action chunk per decision, whereas the best-of-NN variant samples and evaluates 32 chunks at every decision.

We also evaluated all methods on the remaining OGBench tasks of [1] with 10 seeds.

QC-FQLQC (best-of-N)ACH
cube-double43±7 → 100±068±2 → 99±115±6 → 99±1
puzzle-3x3-sparse64±13 → 100±0100±0 → 100±091±3 → 100±0
scene-sparse85±6 → 99±180±4 → 98±196±3 → 99±1

All three methods reach near-perfect success rates on these environments. Together, these results show that ACH performs well across environments, both where adapting the chunk length matters and where a fixed length is sufficient. They are included in Appendix D.9 (Table 9) and Appendix D.7 (Table 7, with all baselines).

R2-2. Comparison with SEAR (W1)

SEAR [2] trains a prefix critic while executing prefixes of random length and achieves strong performance in the online RL setting. To compare its treatment of the chunk length with that of ACH, we transferred it to the backbone of ACH, keeping the policy, the critic architecture, and the Off2On protocol identical to ACH. Following SEAR, this variant executes a prefix of uniformly random length during training, trains the critic on all prefixes with equal weights, updates the actor with the value of the full chunk, and executes a fixed receding horizon HH at evaluation. We train it once per environment and evaluate it with each HH, using 4 seeds. The maximum chunk length on humanoidmaze is 5.

H=1H=3H=5H=10ACH
cube-quadruple-10M0±0 → 27±140±0 → 28±140±0 → 38±131±1 → 45±130±0 → 52±6
humanoidmaze-medium43±12 → 60±1327±8 → 49±1112±3 → 30±4–45±2 → 82±1
puzzle-4x47±5 → 50±1622±6 → 43±1119±7 → 36±1215±4 → 35±1013±3 → 90±8

The results show that the best receding horizon differs across environments. Even with the best horizon chosen for each environment after training, the SEAR-style variant remains below ACH. These results indicate that selecting the chunk length based on the prefix values, during both training and evaluation, contributes substantially to the performance of ACH. We added this comparison as Appendix D.10 (Table 10). We also revised the sentence in Section 5 that the reviewer refers to, so that it states that these methods execute chunks of a fixed length regardless of the state and points to this comparison.

AuthorsOpenReview ↗

Authors' response to reviewer x887 (2/6)

R2-3. On tuned fixed lengths, ReBRAC and FINO, and the learning curves on cube tasks (W1)

Tuned fixed lengths (Table 4). The results in Table 4 show that a fixed length chosen per environment is close to ACH on antmaze-giant, antsoccer-arena, and cube-triple. However, the best fixed length differs across the six environments, and a length that is best in one environment fails in another. For example, H10 performs well on cube-triple but poorly on antmaze-giant and puzzle-4x4. The fixed length therefore has to be searched separately for each environment, which requires a full training run for every candidate. In contrast, ACH uses a single configuration. It matches the best fixed length where one length is sufficient, and clearly exceeds it on cube-quadruple, humanoidmaze, and puzzle-4x4.

ReBRAC and FINO (Table 1). ReBRAC is the best method on antmaze-giant, and FINO is the best on humanoidmaze-medium. However, ReBRAC fails on antsoccer-arena and both cube environments, and FINO performs poorly on both cube environments. In contrast, ACH performs well in all six environments. In the additional environments of [R2-1], both ReBRAC and FINO again fall behind on cube-double, whereas ACH performs well in all three (Table 7).

Learning curves on tasks 1 and 2 of the cube environments (Figure 16). In the cube environments, tasks 1 and 2 are simple pick-and-place tasks, whereas task 3 requires unstacking and task 4 requires cyclic rearrangement. Tasks 1 and 2 are therefore the easier tasks, on which the fixed length of QC is already adequate. Since ACH samples the chunk length during online interaction, its success rate rises more slowly than that of QC in the early stage on these tasks. On the harder tasks of the same environments, such as tasks 3 and 4, QC makes limited progress, whereas ACH learns substantially faster and reaches higher success rates (Table 15). Averaged over the five tasks of each environment, ACH reaches a higher success rate than QC with the same number of online steps in all six environments (Table 1).

Overall, these results show that ACH performs well across all environments with a single configuration, and that its advantage is largest on the harder tasks.

R2-4. Separating the adaptive lengths from the Transformer critic (W2)

We thank the reviewer for suggesting this control. For each environment, we take the best fixed chunk length HH in Table 4 and replace the MLP critic of the backbone with the Transformer critic of ACH, keeping its architecture and hyperparameters. The Transformer critic results use 4 seeds.

best HHFixed HH, MLP criticFixed HH, Transformer criticACH
antmaze-giant110±5 → 74±70±0 → 74±61±1 → 72±2
antsoccer-arena350±3 → 82±21±0 → 66±70±0 → 84±1
cube-quadruple-10M101±0 → 41±30±0 → 43±90±0 → 52±6
cube-triple103±1 → 70±41±0 → 73±81±1 → 67±3
humanoidmaze-medium353±3 → 61±737±6 → 65±245±2 → 82±1
puzzle-4x4114±2 → 54±62±2 → 55±1013±3 → 90±8

The results show that the Transformer critic does not substantially improve the performance of the best fixed chunk length in any environment, and the gap to ACH remains on cube-quadruple, humanoidmaze, and puzzle-4x4. Together with the Random, Uniform, and Greedy ablations, which keep the Transformer critic, and the SEAR-style variant in [R2-2], which also uses the same prefix critic, these results indicate that the gains of ACH come from adapting the chunk length rather than from the critic architecture. They are included in Appendix D.3 (Table 5) and summarized in the main text (Section 4.2, Figure 7).

AuthorsOpenReview ↗

Authors' response to reviewer x887 (3/6)

R2-5. On the values and the SMDP claim (Q1)

Which policy QQ and VV evaluate. The value functions in Eq. (7) evaluate the composite policy that generates a chunk with πθ\pi_\theta and samples its length from the soft πL\pi_L, whereas evaluation uses the greedy length. Proposition 2 (i) in Appendix B.1 of the revised paper shows that, under exact policy evaluation and with the same chunk policy, replacing the soft πL\pi_L by the greedy selection does not decrease the value, so the greedy selection at evaluation is a policy improvement step over the evaluated policy. We added this statement to Section 3.4 of the revised paper.

When Eq. (7) recovers the SMDP Bellman equations. We formalize this in Appendix B.1 of the revised paper. In the first term of Eq. (7), the target Rtn+γn+1V(st+n+1)R_t^n+\gamma^{n+1}V(s_{t+n+1}) for a prefix (st,at:t+n)(s_t,\boldsymbol{a}_{t:t+n}) is computed from the rewards and the state that follow the prefix in the replay data. Under deterministic transitions, these are exactly the rewards and the state obtained by executing the prefix open-loop from sts_t, regardless of whether the logged actions were generated by a single chunk, by several chunks, or by a closed-loop behavior policy. Hence, Eq. (7) recovers the SMDP Bellman expectation equations exactly (Proposition 1), and the open-loop consistency condition of [5] is satisfied by any replay data. Since the OGBench and robomimic tasks in our experiments are simulated without stochastic transitions, the replay data in our experiments satisfy this condition.

With stochastic transitions, the rewards and the state recorded after a prefix in the data are no longer guaranteed to match those obtained by executing the prefix open-loop. For example, when the behavior policy selected the later actions of the prefix after observing the intermediate states, the target can exceed the open-loop value of the prefix. In this case, for Eq. (7) to recover the SMDP equations, it is sufficient for the rewards and the state recorded after each prefix to follow the same distribution as under open-loop execution. This corresponds to the strong open-loop consistency of [5] (Definition 2 in [5]) with zero error. We state this at the end of Appendix B.1 and leave the analysis under stochastic transitions for future work. We also revised the statement in Appendix B that the reviewer refers to, so that it says that the target avoids the off-policy bias when the data are strongly open-loop consistent.

R2-6. On the role of the offline stage (Q2)

To examine what the offline stage contributes, we ran ACH without pre-training. This variant, denoted ACH (Online), starts online learning from scratch with the same offline data in the replay buffer, as in RLPD. ACH (Online) uses 4 seeds.

RLPDACH (Online)ACH (Off2On)
antmaze-giant69±971±872±2
antsoccer-arena64±462±884±1
humanoidmaze-medium29±961±1382±1
puzzle-4x464±1270±2090±8

ACH (Online) performs comparably to or better than RLPD, so ACH can also learn in the purely online setting. However, it is clearly below ACH (Off2On) on antsoccer-arena, humanoidmaze-medium, and puzzle-4x4. Without pre-training, online learning starts with an untrained critic, so the chunk lengths selected in the early stage are not based on reliable values. The offline stage therefore contributes through the pre-trained critic, which is not reflected in the offline success rate. These results are included in Appendix D.8 (Table 8).

Reviewer_hMo7OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper present a RL algorithm ACH, which picks the action-chunk length based on estimated values of candidate prefixes. A causal Transformer estimates the values of all prefixes in a single forward pass. During training, ACH uses those values to sample chunk lengths, and at evaluation it picks the greedy length. The proposed method evaluate on 30 OGBench tasks, four robomimic tasks, and simulated VLA benchmarks.

Critical strengths
  1. The motivation is clear. Chunk length affects both how consistent and how responsive the behavior is, and ACH tackles that trade-off with a value-based criterion. As a reviewer, I also like that chunk selection during training is aligned with how it’s used at execution.
  2. The benchmark coverage is broad: 34 tasks in total, with 10-seed results and confidence intervals for OGBench and robomimic, plus ablations of the selection and training strategies. That’s more careful than a lot of papers in this area.
  3. The VLA results look promising. On top of a frozen GR00T N1.7 base, ACH takes average success from 59.4% to 71.0% on WidowX tasks and from 44.5% to 71.9% on Google Robot tasks in SimplerEnv.
Critical weaknesses
  1. The whole method depends on the critic’s rankings, and didn’t see much analysis of that. ACH picks chunk lengths from Q-value estimates of different prefixes, but it’s not clear how well calibrated those rankings are, or what happens when the critic is wrong.
  2. The VLA validation is simulation only. It covers two SimplerEnv embodiments with four seeds, so I don’t think it shows that adaptive chunking helps on a physical robot.
  3. The deployment overhead is unclear. A value function is evaluated at every decision point to choose the chunk length, but the paper doesn’t report inference latency or throughput compared with fixed-length chunking. So the practical cost of the adaptive selection is still an open question.
Questions for authors
  1. How reliable are the critic’s rankings across candidate chunk lengths? Could you report ranking accuracy, or how sensitive the results are to critic error and the softmax temperature?
  2. Could you report ACH’s inference latency relative to the fixed-length baselines, and check whether the gains hold on a physical robot?
4 comments
AuthorsOpenReview ↗

Authors' response to reviewer hMo7 (1/2)

We thank the reviewer for the time and effort devoted to reviewing our paper. The comments helped us strengthen the analysis of the critic and the practical cost of our method. (The changes are marked in blue in the revised paper.) We address each concern below.

R1-1. On the reliability of the critic's ranking (W1, Q1)

To examine whether the critic ranks the candidate chunk lengths correctly, we compared the critic values with Monte-Carlo (MC) returns. At states visited by the policy after online learning, we estimated the MC return of every prefix of a generated chunk. Since each chunk has h+1h+1 prefixes, we formed all (h+12)\binom{h+1}{2} pairs of prefixes, kept the pairs whose MC returns differ significantly, and measured how often the order of the critic values matches the order of the MC returns. We report this pairwise accuracy over 4 seeds.

TaskCriticAlways longerAlways shorter
cube-triple87±12%65%35%
humanoidmaze-medium77±13%44%56%
puzzle-4x491±8%74%26%

In the above table, "Always longer" denotes the pair ordering assigning the longer prefix a higher return, whereas "Always shorter" is the opposite. Note that random ordering would result in a matching rate of 50%. The result shows that the critic of ACH ranks the prefixes far better than the random ranking and two fixed preferences in all three tasks. In particular, in humanoidmaze, where neither fixed preference dominates, our critic design still yields an order matching rate of 77%, demonstrating that it distinguishes the states in which shorter chunks are better from those in which longer chunks are better rather than following a single preference. These results indicate that the prefix values of our critic provide a meaningful signal for selecting the chunk length at each state. These new results are included in Appendix D.11 (Table 11) and summarized in the main text (Section 4.2, Figure 7).

Furthermore, the curve with the legend Random in Figure 6 corresponds to the limiting case in which the ranking is assumed to carry no information and the executed length is chosen randomly and independently of the values. Its performance is consistently below that of ACH, which indicates that the gains come from the information in the critic's ranking.

Lastly, since the softmax temperature β\beta controls how closely the length selection during training follows the values of the critic, we also examined the sensitivity of ACH to β\beta. We trained ACH with β∈0.1,1,10,100\beta\in\\{0.1, 1, 10, 100\\} on antsoccer-arena and puzzle-4x4, where β=1\beta=1 is the value used in all experiments of the paper. Each result is the average over the five tasks of the environment and 4 seeds.

β=0.1\beta=0.1β=1\beta=1 (default)β=10\beta=10β=100\beta=100
antsoccer-arena1±1 → 83±30±0 → 84±10±0 → 78±30±0 → 67±5
puzzle-4x410±2 → 80±513±3 → 90±84±3 → 90±55±3 → 85±9

The default β=1\beta=1 performs best on both environments, and the performance changes moderately when β\beta is changed by a factor of 10. However, with a large β\beta, the length selection during training becomes close to greedy and explores fewer lengths, which lowers the performance on antsoccer-arena. With a small β\beta, the length selection becomes close to uniform and relies less on the values of the critic, which lowers the performance on puzzle-4x4. We added these results as Appendix D.13 (Table 13).

AuthorsOpenReview ↗

Authors' response to reviewer hMo7 (2/2)

R1-2. On the inference latency in the VLA setting (W3, Q2)

Since the inference cost is important in the VLA setting, we measured it in SimplerEnv on a single NVIDIA RTX 4000 Ada GPU, averaged over the two robots. Decision latency is the time to produce the actions to execute from a single observation. Time per step is the policy computation time per environment step, excluding the simulator.

BaseQCAACQC+AACACH
Decision latency (ms, ↓)76.8134.9188.8250.2138.7
Time per step (ms, ↓)9.917.344.959.536.6

Compared with Base, all other methods have a higher decision latency, since they additionally run the one-step policy on top of the VLA backbone. The decision latency of ACH is slightly higher than that of QC (increase rate 3.8/134.9=2.8%), which shows that selecting the chunk length with the Transformer value function adds little overhead, since our architecture returns the values of all prefixes in a single forward pass. AAC and QC+AAC have a higher decision latency, since they draw 20 samples per decision to select the chunk length. The decision latency of ACH remains below the control periods of both robots (200 ms for WidowX at 5 Hz and 333 ms for the Google Robot at 3 Hz).

For the time per step, ACH calls the backbone more often than QC because it replans more frequently when it selects shorter chunks, which increases its time per step. Nevertheless, since each decision of ACH is faster than that of AAC and QC+AAC, its time per step remains comparable to or lower than theirs, while ACH achieves higher success rates (Figure 4). These results show that ACH achieves its performance gains at a reasonable inference cost.

We added these results to the main text (Figure 5 and Section 4.1) and reported the per-robot results, including the number of backbone calls per environment step, in Appendix D.4 (Figure 12).

R1-3. On validation on a physical robot (W2, Q2)

We agree that validation on a physical robot would further strengthen the practical claims. Since setting up and running real-robot experiments was not feasible for us within the limited discussion period, we instead clarify what the current evidence supports.

First, SimplerEnv [1] was built to reproduce the real WidowX and Google Robot setups in simulation, and that work showed that the success rates measured in it correlate strongly with those measured on the real robots. Second, ACH is applied on top of a frozen GR00T N1.7 without modifying the VLA, and it consistently improves the success rate on both embodiments (59.4% to 71.0% on WidowX and 44.5% to 71.9% on the Google Robot). Third, as shown in [R1-2], the decision latency of ACH is below the control periods of both robots, so the adaptive selection does not prevent real-time control.

We state this point in the Limitations (Appendix A) of the revised manuscript and leave the validation on physical robots as future work.


We thank the reviewer again for the valuable comments and would be happy to provide any further clarification.

References:

[1] Li et al. "Evaluating Real-World Robot Manipulation Policies in Simulation." CoRL 2024.

I will maintain my score based on the author's rebuttal.

AuthorsOpenReview ↗

Authors' response to reviewer hMo7 (follow up)

Thank you for maintaining your positive assessment.

Regarding the concern that our VLA experiments cover only two embodiments, we additionally evaluated ACH on RoboTwin, a simulation benchmark for bimanual manipulation with a dual-arm robot, which differs from the single-arm robots of WidowX and the Google Robot. We used π0.5\pi_{0.5} (Black et al., 2025) as the base VLA, followed the protocol of Section 4.1 otherwise, and report the success rates (%) on four tasks with 95% confidence intervals over 4 seeds.

TaskBaseAACACH
Handover Mic2634±347±5
Place Fan5873±471±6
Place A2B Right4752±559±4
Turn Switch6762±475±3
Average5055±263±2

ACH improves the success rate of the base policy on all four tasks and achieves the highest average success rate. It outperforms AAC on three of the four tasks and performs comparably on place_fan, where the difference is within the confidence intervals. These results are consistent with those on WidowX and the Google Robot. Due to the limited discussion period, we compare ACH with the base policy and AAC, and we will include QC and QC+AAC in the final version.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Contributions and Responses to Reviewers

Dear Area Chair and Reviewers,

As the discussion period closes, we would like to summarize our contributions and our responses to the reviewers' concerns.

1. Contributions. ACH is an offline-to-online RL algorithm that adapts the execution length of an action chunk at each decision. Our main contribution is a value-based criterion for the chunk length, which selects the length from the values of the prefixes of a generated chunk consistently during training and evaluation. To evaluate all prefixes efficiently, we adopt a causal Transformer critic that returns their values in a single forward pass. This allows ACH to balance the uncertainty of predicted future states against the jitter of replanning, which a fixed chunk length cannot do.

2. Empirical evidence. Across 34 tasks on OGBench and robomimic, ACH outperforms fixed-length and adaptive baselines in most environments, with the largest gains on long-horizon manipulation. On VLA models, ACH improves the base policy on both SimplerEnv robots and, in our follow-up, on bimanual RoboTwin tasks with a different base VLA.

3. Revisions addressing the reviewers' concerns. During the rebuttal period, we added:

  • Theoretical analysis of what ACH learns. The critic learns the values of the composite policy, greedy length selection improves the policy, and policy iteration converges to the optimal value. Under a BC constraint, it converges to the constrained optimum, where the best chunk length can depend on the state (Section 3.4, Appendix B.1).
  • Controls for the contribution of adaptive lengths, including the best fixed length with the Transformer critic of ACH, a SEAR-style variant, and best-of-N QC (Appendices D.3, D.9, and D.10).
  • Analyses of the critic, showing that its ranking of prefixes agrees with Monte-Carlo returns and that ACH is robust to the inverse temperature β\beta (Appendices D.11 and D.13).
  • Inference cost in the VLA setting, showing that the decision latency of ACH is within 4 ms of that of QC and below the control periods of both robots (Section 4.1, Figure 5).

These additions address the main concerns regarding the theoretical understanding of the objectives, the contribution of adaptive lengths beyond the critic architecture, the reliability of the critic, and the practical cost. Reviewer PNwf raised the score after our theoretical analysis, and Reviewer hMo7 maintained the positive assessment. For the remaining concerns of Reviewer x887 on the maximum chunk length and the offline performance, we posted a follow-up response near the end of the discussion period, and we kindly ask that it be considered in the final discussion.

We believe the revised paper supports the value of adaptive action chunking for offline-to-online RL, with both theoretical analysis and extensive experiments.

Thank you for your time and consideration.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.