acceptodds
Under review as a conference paper at ICLR 2027

ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network

Abstract

Long-horizon, sparse-reward tasks pose a fundamental challenge for reinforcement learning, since single-step TD learning suffers from bootstrapping error accumulation across successive Bellman updates. Actor-critic methods with action chunking address this by operating over temporally extended actions, which reduce the effective horizon, enable fast value backups, and support temporally consistent exploration. However, existing methods rely on a fixed chunk size and therefore cannot adaptively balance reactivity against temporal consistency. A large fixed chunk size reduces responsiveness to new observations, while a small one produces incoherent motions, forcing task-specific tuning of the chunk size. To address this limitation, we propose daptive hunk ize ctor-ritic (). ACSAC leverages a causal Transformer critic to evaluate expected returns for action chunks of different sizes. At each chunk boundary, it adaptively selects the chunk size that maximizes the expected return, supporting flexible, state-dependent chunk sizes. We prove that the ACSAC Bellman operator is a -contraction whose unique fixed point is the action-value function of the adaptive policy. Experiments on OGBench demonstrate that ACSAC achieves strong performance on long-horizon, sparse-reward manipulation and navigation tasks across both offline RL and offline-to-online RL settings.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, offline-to-online RL, action chunking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_aG3WOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes ACSAC, an offline-to-online RL method which adaptively decides a chunk size with a Transformer-based critic that predicts Q-values for prefixes of action chunks. The policy is extracted via BC rejection sampling. ACSAC consistently outperforms prior offline-to-online RL methods in OGBench.

Critical strengths

S1. Method is straightforward and simple.

  • The method trains a Transformer critic that learns a Q function for the prefixs of the action chunk (similar to [1])
  • Then it extracts the policy with rejection sampling from the BC policy.

S2. Strong theoretical and experimental results.

  • The thoeretical results shows that the proposed bellman operator is a γ\gamma-contraction in a deterministic MDP (Section E), demonstrating theoretical motivation on what this algorithm should converge, and how it is anticipated to work.
  • ACSAC consistently outperforms prior offline and offline-to-online methods.
  • Also, qualitative results (Section 5.3) and ablation studies (Section 5.4) support that adaptive action chunking is important and ACSAC's objective is encouraging adaptive action chunking.

[1] Ge Li et al., TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learning, ICLR 2025

Critical weaknesses

None

Questions for authors

Q1. Why markovian policy is not the optimal policy?

  • I think that it is due to the BC constraints from the markovian policy, but it feels counterintuitive to me that even after online adaptation, the non-markovian policy can be better (as non-markovian policy is always suboptimal due to reactivity)
  • Would it be possible to provide explanation or intuition on why non-markovian policy can be better although the MDP is deterministic and fully-observed?
  • This will help the readers who felt the same "conunterintuitive feeling" from reading the paper.
2 comments
AuthorsOpenReview ↗

Response to Reviewer aG3W

We sincerely thank the reviewer for the positive assessment and the insightful question.

[Q1] Markovian optimality and BC constraints after online adaptation


A: A deterministic and fully-observed MDP has an optimal Markovian policy. As the reviewer suggests, the BC constraints matter here. A Markovian policy fitted to individual actions can miss a pattern in the demonstrations that a policy fitted to action chunks can learn. This advantage can remain even after online adaptation because training continues with the same behavior constraint as online transitions are added to the dataset. Moreover, ACSAC retains reactivity, executing a longer prefix when commitment helps and replanning sooner when reactivity matters.

BC constraints. Even in a fully-observed MDP, a demonstrator’s next action can depend on preceding actions. The OGBench play datasets used for manipulation were collected by open-loop, non-Markovian expert policies[1], producing action sequences that “cannot be well captured by a Markovian behavior constraint”[2]. Fitting full-length action chunks with a flow BC policy lets ACSAC capture these non-Markovian behavior patterns.

Intuition. In the 1D line-world of Lazzati et al.[3], reaching either endpoint succeeds. While an optimal Markovian policy moves toward the nearer endpoint, the demonstrator picks left or right at the start of an episode and keeps moving in that direction. At the same position, successful demonstrations therefore take opposite actions. Sampling from that per-state action distribution at every step can produce a random walk, whereas an action-chunking policy keeps one direction within each chunk and reaches an endpoint sooner on average[3].

Even after online adaptation. Single-step, multi-step, and action-chunking methods all continue training as online transitions are added to the dataset under the same behavior constraints used offline. Because the combined dataset still contains the multi-modal offline demonstrations, a single-step behavior policy continues to fit opposite actions at the same state, whereas ACSAC’s flow BC policy proposes coherent action chunks while the critic selects prefixes across NN candidates. Executing coherent action sequences also helps fixed-chunk policies and ACSAC reach sparse rewards that single-step sampling can miss. Finally, a multi-step TD target alone leaves single-step actor execution unchanged, whereas a chunked critic conditions on the executed action sequence to back up multi-step rewards without the off-policy bias of nn-step returns[2]; on OGBench, all Q-chunking methods outperform their 1-step TD and nn-step counterparts[2].

We hope this explanation helps readers who share the same counterintuitive feeling, and we are happy to provide any further clarification.

References


[1] Seohong Park et al., OGBench: Benchmarking Offline Goal-Conditioned RL, ICLR 2025.

[2] Qiyang Li et al., Reinforcement Learning with Action Chunking, NeurIPS 2025.

[3] Filippo Lazzati et al., Why Does Action Chunking Improve Behavioral Cloning Performance in Robotic Control?, arXiv 2026.

Thank you for the clarification, I'll maintain the positive score.

Reviewer_5Ac9OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

ACSAC is an adaptive action-chunking method evaluated on OGBench in offline and offline-to-online reinforcement learning settings. It uses a causal Transformer critic to jointly estimate Q-values for all prefixes of an action chunk, trained with multi-step TD targets at each prefix length. A policy trained through flow-matching behavior cloning generates multiple candidate chunks. ACSAC then jointly selects a candidate and its execution horizon by maximizing the predicted prefix Q-value, executes the selected prefix, and replans.

Critical strengths

None

Critical weaknesses
  1. Limited validation of the reliability of cross-horizon selection ACSAC’s central mechanism relies on comparing learned prefix Q-values across horizons through a hard argmax. Theorem 1 establishes comparability for true values, but does not ensure accurate cross-horizon rankings under function approximation. This leaves an important gap between the theoretical justification and the implemented selection rule, limiting my confidence that the selected horizons reliably reflect differences in expected return rather than estimation error.

  2. Limited isolation of the adaptive-selection benefit across environments Table 8 isolates horizon selection criteria using identical trained checkpoints. However, the experiment is restricted to cube-triple. Since ACSAC combines a causal Transformer critic, multi-horizon supervision, adaptive bootstrap targets, and adaptive execution, the existing evidence does not establish whether adaptive execution explains the improvements consistently across the benchmark. This limits the strength of the paper’s central attribution of its gains to state-dependent chunk-size selection.

Questions for authors
  1. Greedy selection is well motivated when prefix Q-values accurately estimate comparable returns. Could the authors examine whether predicted return differences across horizons agree with rollout-based differences? This would strengthen the justification for hard argmax selection in the presence of critic estimation error.

  2. Figure 1 and its caption suggest that single-step execution produces incoherent motion, whereas action chunking improves temporal coherence. However, I could not find a direct quantitative evaluation of temporal coherence in the paper. Could the authors define what they mean by temporal coherence and provide empirical evidence comparing single-step, fixed-chunk, and adaptive execution under controlled conditions? If this claim is primarily motivated by prior work, please identify the supporting evidence and qualify the caption to distinguish a possible benefit of chunking from an inherent property.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 5Ac9 (#1)

We sincerely thank the reviewer for the careful review and constructive comments, and address each of them below.

[W1, Q1] Reliability of cross-horizon selection


A: We have tested the reliability of cross-horizon selection in the revision. Please refer to Figures 10–11 in Appendix D.8. Across 150 states, predicted return differences across horizons agree in sign with rollout-based differences for 87.5% of horizon pairs. Scaling the five mean rollout returns at each state from 0 (worst) to 100 (best), the horizon with the highest learned prefix Q-value scores 94.0 on average, compared with 7.3 for the lowest. These results support hard argmax selection because the selected horizons reliably reflect differences in expected return rather than estimation error.

What is compared. Under Theorem 1, true prefix values share a common return scale because the same policy continues after every prefix, and Figure 3 checks calibration on the prefixes ACSAC selects. To test cross-horizon rankings under function approximation directly, we fix the state and candidate chunk so that horizon comparison is isolated from candidate-chunk selection. For each h∈{1,…,5}h\in\{1,\dots,5\}, we execute the first hh actions before following ACSAC for the rest of the rollout, and compare QϕQ_\phi with the mean discounted return from that state over 50 rollouts (Appendix D.8).

State-dependent horizon selection (Figure 10). Matching the task phases in Figure 4 on cube-triple-task4, learned prefix Q-values and mean rollout returns peak together at h=1h=1 in a sensitive state (A), h=3h=3 at an intermediate state (B), and h=5h=5 in a stable phase (C), identifying both the highest- and lowest-return horizons in all three states. At C, QϕQ_\phi lies below the mean rollout return at every horizon yet still peaks at h=5h=5: because hard argmax compares prefixes at the same state, any shared state-level estimation offset cancels across horizons.

Predicted differences and rankings (Figure 11). While shared state-level offsets cancel, horizon-dependent estimation errors could still affect rankings. Across all 1,500 horizon pairs from 150 replanning states (5 cube-triple tasks, 4 seeds), predicted return differences across horizons agree in sign with rollout-based differences for 1,312 pairs (87.5%), with binned means increasing monotonically along y=xy=x (Figure 11, left). Ranking the five horizons by QϕQ_\phi at each state, mean normalized rollout return decreases monotonically across ranks, from 94.0 at rank 1 (hard argmax) to 7.3 at rank 5 (Figure 11, right), supporting hard argmax selection in the presence of critic estimation error.

AuthorsOpenReview ↗

Response to Reviewer 5Ac9 (#2)

[W2] Isolation of the adaptive-selection benefit across environments


A: We have extended Table 8 beyond cube-triple to three manipulation and three navigation domains in the revision, with three evaluated after offline training and three after online fine-tuning (Table 8 in Appendix D.4, shown below). Because all variants use identical trained checkpoints, Table 8 isolates adaptive execution from the causal Transformer critic, multi-horizon supervision, and adaptive bootstrap targets. Adaptive execution improves success by 11–29 percentage points over the best fixed chunk size in each domain and by 20–55 points over uniform selection.

Controlled execution comparison. For each task and seed, all rules share the exact same flow BC policy and causal Transformer critic, scoring all HH prefixes of NN candidate chunks in one forward pass to match candidate budgets and critic computation (Appendix D.4). Fixed-length selection uses the same hh at every state, and uniform selection samples hh uniformly at each replanning state; both select the highest-value length-hh prefix among the NN candidates, whereas adaptive execution selects both chunk and length.

Table 8 (Appendix D.4): Effect of chunk size selection across domains (success %, 5 tasks × 4 seeds × 50 episodes; -: h>Hh>H).

DomainStageAdaptiveh=1h=1h=2h=2h=3h=3h=4h=4h=5h=5Uniform
cube-tripleOnline98323754696343
cube-quadrupleOnline73314050595441
antmaze-giantOnline87374052546151
cube-doubleOffline85716661656263
antsoccer-arenaOffline52414037313432
humanoidmaze-mediumOffline66534740--44

Across environments. The best fixed length shifts from h=1h=1 after offline training to h=4h=4–55 after online fine-tuning across the six domains. Even tuning a fixed length per domain yields lower success than adaptive execution, and state-independent uniform selection falls below the best fixed length in every row. The improvements consistently come from state-dependent chunk-size selection.

Relation to Table 1 and Figure 5(c). In Table 8, fixed h=1h=1 matches or exceeds single-step BFN (N=4N=4, Tables 1 and 4), while fixed h=5h=5 is similar to or below QC because ACSAC samples N=4N=4 chunks (N=8N=8 on cube-quadruple) rather than QC’s N=32N=32 (Table 4). With this smaller chunk budget, evaluating all NHNH prefixes in one forward pass lets adaptive execution match QC on cube-quadruple and outperform it on the other five domains. In Figure 5(c), we also replace the MLP critics in QC and QC-FQL with ACSAC’s causal Transformer critic trained from scratch under fixed-chunk backups and execution (QT-QC and QT-QCFQL), and both remain well below ACSAC on cube-triple. Together, Figure 5(c) (end-to-end fixed-chunk training with the same critic backbone) and Table 8 (identical trained checkpoints with multi-horizon supervision and adaptive bootstrap targets held fixed) establish that ACSAC’s gains consistently require state-dependent chunk-size selection.

AuthorsOpenReview ↗

Response to Reviewer 5Ac9 (#3)

[Q2] Temporal coherence under single-step, fixed-chunk, and adaptive execution


A: By temporal coherence we mean that consecutive executed actions follow one temporally extended motion, without pauses or jittery motions, as motivated by prior action-chunking work. In the revision, we compare single-step, fixed-chunk, and adaptive execution under controlled conditions on cube-triple-task4 using identical trained checkpoints. Please refer to Figure 12 in Appendix D.9. On this task, fixed-chunk and adaptive execution both sustain motion over five steps, with greater mean end-effector displacement than single-step execution. We have also qualified the Figure 1 caption to state this as a possible benefit (“can produce incoherent motions”; “can improve motion coherence”).

Definition and supporting evidence in prior work. Prior work defines this property consistently and provides the observations motivating Figure 1. QC describes temporally coherent actions as actions that “resemble temporally extended skills,” showing pauses in single-step BFN trajectories and higher five-step displacement for QC (Section 5.4, Figures 5, 8–9)[1]. Diffusion Policy contrasts “temporal action consistency” with “jittery actions that alternate between the two valid trajectories” and notes that single-step policies can overfit to demonstration pauses (Figure 5, left)[2], while ACT links non-chunked policies to “pauses in the middle of a demonstration”[3].

Controlled comparison (Figure 12). Using identical trained checkpoints on cube-triple-task4 (sharing the proposal policy, critic, and maximum chunk size H=5H=5), we vary only the execution rule: single-step (h=1h=1), fixed-chunk (h=Hh=H), or adaptive (h=h⋆h=h^\star). Following QC[1] (Section 5.4, Figure 5), we measure the mean Euclidean distance between 3D end-effector positions recorded every five steps (Appendix D.9), which decreases when pauses or reversals leave the end effector close to where it started. Over 500 environment steps (Figure 12, top), the single-step trajectory forms a dense cluster of positions where motion pauses, whereas fixed-chunk and adaptive execution spread over a wider region as the end effector continues to move. Across evaluation steps (Figure 12, bottom), five-step displacement under single-step execution decreases steadily with repeated pauses, while adaptive execution stays close to fixed-chunk execution at consistently higher displacement, showing that adaptive replanning preserves sustained motion.

We hope these results address the reviewer's concerns, and we are happy to provide any further clarification.

References


[1] Qiyang Li et al., Reinforcement Learning with Action Chunking, NeurIPS 2025.

[2] Cheng Chi et al., Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, IJRR 2025.

[3] Tony Z. Zhao et al., Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware, RSS 2023.

Reviewer_WpN3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Adaptive Chunk Size Actor-Critic (ACSAC) for offline and offline-to-online reinforcement learning. At each replanning state, a flow behavior-cloning policy generates candidate action chunks, and a causal Transformer critic evaluates every candidate prefix. The method jointly selects the candidate and prefix length with the highest estimated Q-value, executes that prefix, and replans. The critic is trained using multi-step targets at all prefix lengths. The paper provides a Bellman contraction result and evaluates ACSAC on 40 OGBench tasks, reporting improvements over single-step, multi-step, and fixed-length chunking baselines, together with calibration analyses and ablations.

Critical strengths
  1. The method addresses a meaningful control trade-off. Selecting execution length according to the state provides a practical way to balance temporally coherent actions with responsiveness. The joint candidate-and-prefix selection rule is straightforward, and the paper explicitly formulates the corresponding value-learning objective.

  2. The empirical results provide encouraging evidence. The evaluation covers both manipulation and navigation in offline and offline-to-online settings. The Transformer-backbone ablation and the fixed/random-horizon comparisons in Appendix D.4 provide useful evidence that the reported improvement involves more than replacing the critic architecture.

Critical weaknesses
  1. The novelty is insufficiently established against the closest prior work. The most consequential omission is Adaptive Action Chunking via Multi-Chunk Q Value Estimation (ACH, May 2026). ACH already uses a causal Transformer to estimate prefix-conditioned Q-values and adapt execution length during training and inference. ACSAC differs through its joint maximization over candidate chunks and prefix lengths, its flow BC proposal policy, and its bootstrap construction, but the paper needs to explain why these differences constitute a meaningful advance and demonstrate their practical benefit through a direct comparison. More broadly, critic-guided best-of-N chunk selection is already central to QC/DQC, T-SAC already uses a causal Transformer critic, and SEAR already combines policy chunking with a multi-horizon Transformer critic. Although several of these components are acknowledged, the contribution should be positioned narrowly around the particular adaptive selection and backup rule.

  2. The evidence does not yet sufficiently characterize when adaptive selection helps or how reliably it operates. QC is already included, and Appendix D.4 compares adaptive selection with fixed-length best-of-N and uniform-length selection using the same trained checkpoints. These are relevant controls, but the latter analysis is limited to cube-triple after online fine-tuning. Broader comparisons would help establish whether the benefit consistently comes from state-dependent execution. Selecting among horizon-specific Q-values also introduces sensitivity to estimation errors and maximization bias. Averaging comparable estimates can reduce variance when their errors are imperfectly correlated, but averaging values for different horizons can change the quantity being estimated. The stability–flexibility trade-off therefore needs a carefully defined, controlled comparison. Finally, the qualitative rollout examples do not show how execution lengths evolve during training or whether selection concentrates near one-step or maximum-length execution across tasks.

Questions for authors
  1. Closest prior work: How does ACSAC differ technically from ACH, particularly in candidate generation, execution-length selection, and value targets? Can you provide a direct comparison under the same training and evaluation protocol, and identify which differences account for any improvement?

  2. Selection mechanism and stability: Can you extend the matched fixed-length best-of-N comparisons beyond the final cube-triple checkpoints, controlling the proposal policy, critic, candidate count, and computation? A DQC comparison would also help establish the benefit beyond existing chunk-selection methods. If an aggregate-score alternative is evaluated, what quantity does it estimate, and how does its estimation stability compare with horizon-specific selection?

  3. Execution lengths during training: Can you report the mean and distribution of executed chunk lengths at checkpoints throughout offline training and online fine-tuning, across representative tasks and seeds? In particular, please report the frequencies of one-step and maximum-length execution. Together with the existing fixed/random-horizon controls, these results would help distinguish sustained adaptive behavior from concentration at either endpoint.

5 comments
AuthorsOpenReview ↗

Response to Reviewer WpN3 (#1)

We sincerely thank the reviewer for the careful review and constructive comments, and address each of them below.

[W1, Q1] Direct comparison with ACH and component replacements


A: ACH and ACSAC are independent, concurrent work. Nevertheless, we have cited ACH and added a direct comparison under the same training and evaluation protocol in the revision (Table 9 in Appendix D.6, shown below). ACSAC outperforms ACH by 8–52 points offline on five domains and by 9–32 points online on all six.

Both ACH and ACSAC start with the same motivation, that the execution length of an action chunk should adapt to the state, and therefore need a network architecture that takes action chunks of different lengths as input and estimates their prefix-conditioned Q-values. For this, both adopt the causal Transformer critic of TOP-ERL[1], and both learn a flow BC policy over action chunks as in QC. However, ACSAC differs from ACH in value targets, candidate generation, and execution-length selection, and replacing any of the three components lowers success relative to full ACSAC.

Direct comparison. Table 9 compares full ACH (arXiv Table 1, 10 seeds) and ACSAC (8 seeds: four rows from Table 7 plus cube-quadruple-10M and puzzle-4x4 trained to match ACH's datasets) across all six OGBench domains reported by ACH. The last three columns replace one component of ACSAC at a time with its ACH counterpart, holding the other two components and shared hyperparameters fixed (4 seeds; Appendix D.6). Every replacement lowers online success on all six domains and offline success on all five domains above 0%.

Table 9 (Appendix D.6): Comparison with ACH and single-component replacements in ACSAC (offline → online success %, 1M steps per phase, 5 tasks × 50 episodes).

DomainACHACSACACH Value targetsACH Candidate generationACH Execution-length selection
cube-triple1 → 6718 → 990 → 585 → 773 → 32
antmaze-giant1 → 720 → 870 → 540 → 720 → 30
antsoccer-arena0 → 8452 → 9334 → 7831 → 8129 → 64
humanoidmaze-medium45 → 8265 → 9238 → 6942 → 7320 → 51
cube-quadruple-10M0 → 528 → 620 → 364 → 460 → 12
puzzle-4x413 → 9046 → 10029 → 6038 → 8034 → 57

Value targets (bootstrap construction). ACH reuses TOP-ERL's critic loss[1], bootstrapping each prefix Q-value from a learned state value fitted to a proposed chunk's prefix Q-values under its length policy, without a convergence or fixed-point result. ACSAC instead bootstraps directly from the joint maximum over candidate chunks and prefix lengths (Section 4.2, Eqs. 4–5); with ACH's value targets, online success drops by 15–41 percentage points.

Candidate generation (flow BC proposal policy). Like QC-FQL, ACH distills its flow BC policy into a one-step actor trained with Q-value gradients and an environment-specific behavior cloning coefficient, and samples one chunk from this actor. Like QC, ACSAC instead samples NN chunks directly from its flow BC policy, with no actor Q-value gradients or behavior cloning coefficient (Section 4.3, Eq. 7); with ACH's candidate generation, online success drops by 12–22 percentage points.

Execution-length selection (joint maximization over candidate chunks and prefix lengths). ACH's chunk length selection policy πL\pi_L chooses only among the HH prefixes of a single chunk (softmax in training, greedy at evaluation), whereas ACSAC uses rejection sampling to jointly maximize over NN candidate chunks and HH prefix lengths (Section 4.3, Eq. 8). Replacing ACSAC's selection rule with ACH's reduces the search space from NHNH candidate prefixes to HH: whereas ACH uses Q-value gradients to improve the actor before proposing a single chunk, ACSAC samples from the flow BC policy and relies on rejection sampling, so a single sampled chunk prevents selecting a higher-value prefix from another chunk. Online success drops by 29–67 percentage points, the largest drop in all six domains.

Joint design. Using the same joint maximum for execution and bootstrapping yields a γ\gamma-contraction whose unique fixed point is the action-value function of the executed policy (Theorem 2, Appendix E.3). Replacing one component in isolation can break this correspondence between policy and backup while separating ACH's own coupled components (see Section 2 and Appendix D.6).

AuthorsOpenReview ↗

Response to Reviewer WpN3 (#2)

[W1] Relation to QC/DQC, T-SAC, and SEAR


A: Our contribution extends beyond the particular adaptive selection and backup rule. By generalizing QC’s critic-guided best-of-NN chunk selection and the use of a causal Transformer critic in TOP-ERL[1], we develop a new adaptive action-chunking algorithm and establish its theoretical guarantees. ACSAC brings state-dependent chunk selection to offline and offline-to-online RL, achieving higher average success than fixed-chunk baselines on both manipulation and navigation tasks (Table 1).

  1. Critic-guided best-of-NN chunk selection. BCQ[2], EMaQ[3], and diffusion-based SfBC[4] generate candidates from a behavior policy and select among them by Q-value; QC extends best-of-NN selection to flow policies and action chunks, and DQC decouples policy and critic chunk sizes while keeping execution length fixed across states. Rather than choosing among NN fixed-length chunks, ACSAC selects both a chunk and its execution length in a “joint search over NHNH candidates indexed by (n,h)(n, h)” (Section 4.3), increasing the search space without sampling more chunks. For example, if a chunk's first action has high value but later actions would lower the return, ACSAC can execute the first action and replan.

  2. Causal Transformer critic. TOP-ERL[1] introduced the causal Transformer critic over action chunks for episodic RL. Online T-SAC pairs this critic with a single-step squashed Gaussian actor, while online SEAR combines policy chunking with a multi-horizon Transformer critic, training a squashed Gaussian actor with Q-value gradients and state-independent replanning (random during data collection, fixed at evaluation). In offline and offline-to-online RL, ACSAC instead pairs a flow BC proposal policy (without actor Q-value gradients) with critic-guided selection of both chunk and execution length, generalizing EMaQ’s expected-max Q operator from NN actions to NHNH candidate prefixes.

ACSAC uses the same adaptive selection and backup for offline training and online fine-tuning. Theorem 1 shows that prefixes of different lengths share a common return scale, and Theorem 2 proves that the backup is a γ\gamma-contraction whose unique fixed point is the action-value function of the executed policy (Section 4.2, Appendix E).

AuthorsOpenReview ↗

Response to Reviewer WpN3 (#3)

[W2, Q2] Matched fixed-length best-of-NN and DQC comparisons


A: We have extended the matched fixed-length best-of-NN comparisons beyond cube-triple to six domains in the revision (Table 8 in Appendix D.4, shown below). Using the same trained checkpoints, selecting the chunk size at each state improves success over the best fixed length by 11–29 points across three domains after offline training and three after online fine-tuning, and over uniform-length selection by 20–55 points. In a separate comparison, ACSAC outperforms DQC on all five manipulation domains, offline and online.

Matched execution rules. At each replanning state, all rules sample N=4N=4 chunks (88 on cube-quadruple, vs. QC's N=32N=32 in Table 4) from the same flow BC proposal policy and score all HH prefixes in one critic pass, controlling the proposal policy, critic, candidate count, and computation (Appendix D.4). Fixed-hh uses a constant hh and Uniform samples h∼Unif([H])h\sim\mathrm{Unif}([H]), each selecting the best length-hh prefix among the NN chunks; ACSAC selects both chunk and length.

Table 8 (Appendix D.4): Effect of chunk size selection across domains (success %, 5 tasks × 4 seeds × 50 episodes; -: h>Hh>H).

DomainStageAdaptiveh=1h=1h=2h=2h=3h=3h=4h=4h=5h=5Uniform
cube-tripleOnline98323754696343
cube-quadrupleOnline73314050595441
antmaze-giantOnline87374052546151
cube-doubleOffline85716661656263
antsoccer-arenaOffline52414037313432
humanoidmaze-mediumOffline66534740--44

When adaptive selection helps. Across the three manipulation and three navigation domains, the best fixed length shifts from h=1h=1 offline to h=4h=4–55 online. Adaptive selection exceeds every fixed length, while state-independent Uniform falls below the best fixed length in every domain. The benefit consistently comes from state-dependent execution.

DQC comparison. DQC uses fixed-length best-of-NN chunk selection with a policy chunk size tuned per domain and shared across all states. On the five manipulation domains of Table 1, AQC[5] (Table 1) evaluates the best DQC configuration per domain under the same reward-based protocol; ACSAC outperforms DQC by 7–70 points offline and 3–98 points online.

Table: Comparison with DQC on manipulation domains (offline → online success %, 5 tasks × 4 seeds).

DomainDQCACSAC
puzzle-3x3-sparse86 → 97100 → 100
scene-sparse56 → 89100 → 100
cube-double15 → 5585 → 100
cube-triple1 → 017 → 98
cube-quadruple2 → 209 → 73

[W2, Q2] Aggregate score and estimation stability


A: For the aggregate-score alternative, Qˉϕ(st,at:t+H)=1H∑h=1HQϕ(st,at:t+h)\bar Q_\phi(s_t,a_{t:t+H})=\frac1H\sum_{h=1}^H Q_\phi(s_t,a_{t:t+h}) estimates Eh∼Unif([H])[Qπ(st,at:t+h)]\mathbb E_{h\sim\mathrm{Unif}([H])}[Q^\pi(s_t,a_{t:t+h})], the expected discounted return of executing a uniformly sampled prefix of a chunk and following π\pi (Theorem 1). Averaging improves estimation stability (11–12% lower error SD and smaller maximization bias), while horizon-specific selection improves success by 5–13 points offline and 22–31 points online.

Controlled comparison. At each replanning state, both rules score NN chunks and N×HN\times H prefix Q-values (5 tasks, 4 seeds). Horizon-specific selects (n⋆,h⋆)(n^\star,h^\star) maximizing QϕQ_\phi (Eq. 8); Aggregate selects the chunk maximizing Qˉϕ\bar Q_\phi and samples h∼Unif([H])h\sim\mathrm{Unif}([H]) (unlike Uniform above, which samples hh first and uses one head). Across matched states, error is each rule’s predicted value minus its 50-rollout return under π\pi, on the selected prefix (Horizon-specific) or the selected chunk’s HH-prefix mean (Aggregate).

Table: Horizon-specific vs. aggregate-score selection.

cube-tripleHorizon-specificAggregate
Mean error+1.2+0.8
Error SD18.616.4
Offline → online (%)17 → 9812 → 67
antsoccer-arenaHorizon-specificAggregate
Mean error+1.5+0.9
Error SD26.523.6
Offline → online (%)52 → 9339 → 71

Stability–flexibility trade-off. Errors across targets GhG_h are imperfectly correlated while sharing prefix actions and critic parameters (Lemma E.15, Remark E.16), so averaging reduces only horizon-specific error and lowers overestimation by 0.4–0.6 return units. While Qˉϕ\bar Q_\phi ranks chunks better than single-head Uniform above, it averages out differences across prefixes within a chunk: if a chunk’s first action has high value (h=1h=1) but later actions lower the return, averaging lowers the score and cannot replan after one step, whereas horizon-specific selection executes h=1h=1 and replans.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Kind Reminder and Summary of Revisions Before Discussion Deadline

We sincerely thank the Area Chair and all reviewers for their constructive comments. On November 15 (UTC/AoE), we posted point-by-point responses and uploaded a revised PDF (changes marked in blue). While Reviewer aG3W has kindly responded and confirmed their positive score, as the discussion deadline (November 18 AoE) is approaching, we would greatly appreciate hearing from Reviewer WpN3 and Reviewer 5Ac9 on whether our responses and revisions resolve their concerns.

Below is a brief summary of our main responses and manuscript revisions:

  1. Direct comparison with ACH and relation to prior work (Table 9, Appendix D.6). We cited concurrent work ACH, clarified ACSAC's positioning relative to QC/DQC, TOP-ERL, T-SAC, and SEAR, and added a six-domain comparison under the same protocol: ACSAC outperforms ACH by 8–52 points offline on five domains and by 9–32 points online on all six, and improves online success by 15–41, 12–22, and 29–67 points over single-component replacements in value targets, candidate generation, and execution-length selection.
  2. State-dependent chunk-size selection across domains and DQC comparison (Table 8, Appendix D.4). Extending our matched-checkpoint evaluation to six manipulation and navigation domains across offline and online stages, adaptive execution improves success by 11–29 points over the best fixed chunk size in each domain and by 20–55 points over uniform-length selection, while outperforming DQC on all five manipulation domains by 7–70 points offline and 3–98 points online.
  3. Reliability of cross-horizon selection and aggregate-score analysis (Figures 10–11, Appendix D.8). Across 1,500 horizon pairs from 150 replanning states, predicted return differences across horizons agree in sign with rollout-based differences for 87.5% of pairs and rank-1 (hard argmax) achieves a 94.0/100 mean normalized rollout return; horizon-specific selection also improves success by 5–13 points offline and 22–31 points online over an aggregate-score alternative averaging prefix Q-values.
  4. Execution lengths throughout training (Figure 9, Appendix D.7). We added the mean and distribution of executed chunk lengths throughout offline training and online fine-tuning, showing that mean execution length rises over training while all five lengths remain in use, with intermediate lengths (h=2,3,4h=2,3,4) selected more often in total than h=5h=5 rather than concentrating at either endpoint.
  5. Definition and quantitative evaluation of temporal coherence (Figure 12, Appendix D.9). We defined temporal coherence based on prior action-chunking work, qualified the Figure 1 caption, and compared single-step, fixed-chunk, and adaptive execution using identical trained checkpoints, confirming that adaptive execution avoids single-step pauses and sustains five-step end-effector displacement close to fixed-chunk execution.

We remain happy to provide any further clarification before the discussion period concludes.

AuthorsOpenReview ↗

Summary of Core Contributions and Thank You to Reviewers

Dear Reviewers and Area Chair,

As the author-reviewer discussion period comes to a close, we sincerely thank the Area Chair and Reviewers aG3W, WpN3, and 5Ac9 once again for your time and constructive comments, which helped strengthen our analyses and manuscript. Below, we briefly summarize our core contributions and main findings:

  • Adaptive action chunking on a common return scale: Rather than fixing one chunk size across all states to trade off reactivity against temporal consistency, ACSAC adaptively selects the chunk size at each replanning state. A causal Transformer critic evaluates all candidate prefixes from a flow BC policy in one forward pass without causal leakage on a common return scale (Theorem 1), and the same joint argmax over candidate chunks and prefix lengths drives execution and multi-step TD backups with guaranteed Bellman contraction to the adaptive policy's value function (Theorem 2).

  • Consistent offline and offline-to-online gains across 40 tasks: Across eight long-horizon, sparse-reward manipulation and navigation domains on OGBench, ACSAC achieves the highest average offline and online success in both manipulation (62% to 94%) and navigation (39% to 91%) (Table 1) using half the parameters (2.46M) and only 4–8 candidate chunks (vs. QC's 32), while outperforming DQC and concurrent work ACH under matched protocols.

  • State- and stage-dependent adaptation: Effective execution horizons vary across both task phases (shorter prefixes at sensitive states and longer prefixes during stable phases) and training stages (shifting from short reactive prefixes offline toward intermediate and longer prefixes online without concentrating at either endpoint; Figure 9), yielding 11–29 point gains over each domain's best fixed chunk size on identical checkpoints across six domains (Table 8) and 76.1–87.5% cross-horizon rollout sign agreement across manipulation and navigation.

Thank you once again for your engagement, feedback, and consideration of our work.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.