acceptodds
Under review as a conference paper at ICLR 2027

Chunking the Policy: Extending the Policy-Update Horizon with a Compact Transformer

Abstract

We introduce HOPS (Horizon-Extended Optimization over Policy-Generated Sequences), an off-policy actor–critic method that independently controls the policy-update horizon while keeping interaction-time prediction and execution horizons fixed. A compact Transformer autoregressively composes action chunks of length , a task-dependent hyperparameter, into longer sequences for actor optimization while retaining short prediction and execution horizons during interaction. In the default Meta-World configuration, : actor updates optimize sequences of up to sixteen actions, while interaction predicts and executes (default: 4) actions before replanning. Twin prefix-conditioned critics provide value gradients across the generated sequence. We train separate agents online from scratch on all 50 Meta-World ML1 tasks and dense-reward FANCY GYM Box Pushing, using final-timestep success for evaluation. HOPS achieves a mean task-wise success IQM of on Meta-World at environment interactions, compared with for T-SAC, the previous State-of-the-art baseline. On Box Pushing, HOPS reaches a success IQM of 0.941 at interactions, compared with 0.637 for T-SAC. Controlled ablations on a 20-task Meta-World subset vary the maximum policy-update horizon while holding the interaction horizons and critic training fixed. Extending the maximum update horizon from four to sixteen improves aggregate performance, with task-dependent effects. These results support extending the temporal scope of actor optimization through chunk-autoregressive generation while retaining short action chunks for interaction.

Primary area:
reinforcement learning
Keywords:
SAC, Soft Actor-Critic, Transformer, Transformer Critic, Transformer Policy, Action Chunking

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_ZXACOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a new off-policy AC method denoted HOPS which extends T-SAC. It uses a causal transformer as an actor generating sequences of actions. The sequences are scored by a critic (also a transformer) which ranks and chooses the best actions from the diverse set of sequences. Results show a clear advantage for HOPS over T-SAC on 50 Meta-World ML1 tasks and Box Pushing.

Critical strengths
  • The paper is well-organized and easy to follow.

  • The paper reports strong results across all 50 Meta-World tasks plus Box Pushing

Critical weaknesses
  • The paper doesn't report a one-step actor update (Hupd=1H_{\text{upd}}=1) variant of HOPS. As a result, the value from the transformer long update horizon is unclear and unsupported. If the authors can demonstrate the benefits of a longer policy-update horizon over Hupd=1H_{\text{upd}}=1, I will happily consider pushing my score upward.

  • It is unclear whether T-SAC was rerun in the HOPS pipeline or imported from Tian et al. (2026). This is important because HOPS changes several T-SAC settings (that its own ablations show matter: hard target copies, α=0\alpha=0, learning rate, buffer size).

  • Missing baselines for comparison: SEAR (Nagy et al., arXiv March 2026) cited not compared against, GPM (Zhang, Xu and Yu, ICLR 2022).

  • The theoretical analysis is a standard variance identity under untested assumptions, and it does not explain when or why a longer update horizon helps.

Questions for authors
  1. Was C=4C=4 used for all 50 tasks in Table 1?

  2. Is the horizon effect significant across tasks and seeds, for example under a paired or stratified bootstrap?

4 comments

Thank you for the time you spent evaluating the paper. Your request for a one-step actor control and your questions about baseline provenance and statistical uncertainty helped us test the horizon claim more directly. Your comments also helped us distinguish the empirical findings from the motivation provided by the variance analysis.

Critical Weaknesses

The paper doesn't report a one-step actor update ([…]) variant of HOPS. As a result, the value from the transformer long update horizon is unclear and unsupported. If the authors can demonstrate the benefits of a longer policy-update horizon over […], I will happily consider pushing my score upward.

We added the one-step control on 12 Meta-World tasks (Appendix B.3, Table 8). We fixed generation, execution, and bootstrap length at one action and compared a one-action update with a length sampled uniformly from 1 through 16. With best-of-32, final success changed from 56.0% to 61.6% (+5.6 percentage points, 95% CI [−7.4, 22.4]). With single-sample selection, it changed from 54.2% to 51.9% (−2.3 points [−21.9, 16.1]). Both intervals include zero. The endpoint benefit over a one-step update therefore remains uncertain.

It is unclear whether T-SAC was rerun in the HOPS pipeline or imported from Tian et al. (2026). This is important because HOPS changes several T-SAC settings (that its own ablations show matter: hard target copies, […], learning rate, buffer size).

Table 1 uses the released T-SAC results from Tian et al. (2026). It compares the complete systems, including their different training and selection settings. Appendix B.3 separately reports new T-SAC controls. With one-action execution and single-sample selection, extending T-SAC’s update changed final success from 53.8% to 61.2% (+7.4 points [−15.8, 23.3]). We also evaluated the original T-SAC MLP with longer outputs and best-of-32. These controls do not fully match the HOPS training pipeline.

Missing baselines for comparison: SEAR (Nagy et al., arXiv March 2026) cited not compared against, GPM (Zhang, Xu and Yu, ICLR 2022).

We added a SEAR comparison on the same 12 Meta-World v3 tasks at approximately 2.05M interactions (Appendix B.10). HOPS reached 82.8% final success, compared with 79.6% for SEAR-10 executing four actions (+3.2 points [−17.2, 18.3]). HOPS used best-of-32 and 160 evaluation episodes. SEAR used 20 episodes per execution length, and four was its highest observed setting in a sweep from one through ten. The interval and protocol differences leave superiority unresolved. We have no matched GPM result and make no performance claim against it.

The theoretical analysis is a standard variance identity under untested assumptions, and it does not explain when or why a longer update horizon helps.

We now use the identity to motivate a measurable question about gradient consistency. On 12 tasks, mixed-length updates increased early log⁡10\log_{10} gradient signal-to-noise by 0.111 relative to fixed-four updates (95% interval [0.027, 0.236], Holm-adjusted p=0.027p=0.027). Gradient cosine agreement across sub-batches increased by 0.083 [0.037, 0.134] (p=0.006p=0.006). Both differences were observed during 0.5–1.2M interactions. Neither passed the adjusted threshold later in training (Appendix C, Tables 24–25). These results characterize early optimization under each configuration’s own objective and replay data. Whether they explain improved control remains open.

Questions for Authors

Was C=4C=4 used for all 50 tasks in Table 1?

Yes. All 50 tasks used C=4C=4, with prediction, execution, and bootstrap horizons fixed at four and actor-update lengths sampled uniformly from {4, 8, 12, 16}. Best-of-32 was used during collection and evaluation. We selected C=4C=4 because it had the highest observed aggregate final success at 2M interactions in the earlier six-task, four-seed sweep over C∈{1,2,3,4}C\in\{1,2,3,4\} (Figure 8(b)). We used the same setting across all 50 tasks without per-task retuning.

Is the horizon effect significant across tasks and seeds, for example under a paired or stratified bootstrap?

The task-and-seed bootstrap supports a training-period improvement under the default best-of-32 setting. Across all 50 tasks, we resampled tasks and then seeds within tasks to estimate the difference between fixed-four updates and updates sampled uniformly from {4, 8, 12, 16} (Appendix B.2, Table 7).

With best-of-32, the five-checkpoint training score improved by 4.25 percentage points [0.16, 8.24]. The final-success difference was +1.44 points [−3.34, 6.67]. With single-sample selection, the corresponding differences were −0.10 [−5.3, 5.6] and −0.6 [−7.7, 6.9]. The training-score interaction was +4.4 points [−3.1, 10.9]. The endpoint effect and the dependence of the training-period gain on selection therefore remain uncertain.

Reviewer_ZXACOpenReview ↗

Thank you for the response and the new (honest) results.

W1. I appreciate the one-step control. However, as you state, it does not show a benefit of longer updates over a one-action update. The control also changes the generation chunk to one action, where HOPS reaches 56–62% versus about 80% for the default, so it does not test the main configuration. The strongest remaining evidence is the 50-task training-score gain under best-of-32. The final-success and single-sample differences are not significant, and the training score is a new summary introduced in the rebuttal. Other new results show much larger effects from mean versus min bootstrap reduction (+31.5 points) and from the Transformer actor versus an MLP actor with best-of-32 (+28.5 points). Could it be that HOPS’s advantage comes mainly from these design choices rather than the update horizon?

W2. Thank you for clarifying. Please state this in the table caption.

W3. The SEAR comparison is helpful, though inconclusive. Please also discuss GPM in related work.

W4. Thank you. The early-training gradient diagnostics are a reasonable use of the identity, and the claims are now appropriately scoped.

Overall. The paper’s central claim, stated in its title and abstract, is that extending the policy-update horizon improves learning. After the rebuttal, the evidence supports only a narrow version of this claim: an early-training benefit under best-of-N selection. There's no significant gain in final success and no clear benefit over a one-action update. The paper’s other stated contributions, the horizon framework and the HOPS system with its strong results on all Meta-World tasks and Box Pushing, are well supported. Weighing these, I maintain my score of 3, but it is a low 3. I would give 2.5 if that was possible. I suggest revising the title and abstract so the central claim matches the evidence.

Thank you for suggesting that the title and abstract should reflect what the controls establish. We have followed that suggestion. The revised title is A Chunk-Autoregressive Actor for T-SAC, and we withdraw the claim that longer actor updates improve learning. Your observation about the one-action control is important. It uses a one-action generator, so it leaves open what would happen if we retained the default four-action generator and changed only the update length to one. The five-checkpoint training score was also introduced during rebuttal. We report it as exploratory.

Thank you also for pointing out where readers need the baseline provenance. Appendix J identifies the source. We have added the GPM in the related work.

Reviewer_pPj6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HOPS is an off-policy actor-critic method that combines transformer-based critic (with causal masking) with a transformer-based policy (which outputs action chunks); the algorithm allows practitioners to increase the policy update horizon while keeping the execution horizon different. Furthermore, they ablate over different parts of their architecture. The method achieves SOTA results on metaworld and FANCY GYM.

Critical strengths
  • Strong results across benchmarks. Meta-World mean task-wise IQM improves over T-SAC, alongside strong Box Pushing results. Extensive baseline comparisons.
  • A simple, but effective method, useful ablations. The actor optimizes longer action sequences, while retaining short interaction chunks for data collection and inference.
Critical weaknesses
  • Incremental novelty. The method combines established prefix critics and autoregressive policies; advantage over closely related SEAR remains untested.
  • Unclear design rationale and attribution. Inter-chunk context detachment, uniform horizon sampling, and bootstrap averaging lack sufficient justification. The dynamics of these different choices and their effects is relatively understudied (in prior works as well).
Questions for authors
  1. Why detach inter-chunk context? How does full backpropagation affect performance and stability?

  2. Horizon sampling: Why sample uniformly rather than always use 16 actions? What is the uncertainty in the gain over a maximum horizon of 4?

  3. Bootstrapping: What motivates averaging bootstrap values across prefix lengths and critics, and choosing H_target = 4? How do these choices affect value estimation and performance?

  4. Attribution: How does HOPS compare with an MLP actor and SEAR under matched horizons, candidate selection, and training settings?

  5. Could you provide a controlled study linking prefix averaging, horizon sampling, and context detachment to optimization behavior? For example, how do these choices affect gradient variance, critic overestimation, and training stability, and do these effects explain the performance gains?

  6. What motivates the entropy-free default?

Happy to raise my score if these concerns are addressed.

4 comments

Thank you for the careful review and for asking how the design choices affect optimization. Your suggestions led us to test full backpropagation, distinguish the actor and bootstrap objectives more clearly, and compare HOPS directly with SEAR. These changes improved both the technical explanation and the empirical evaluation.

Critical Weaknesses

Incremental novelty. The method combines established prefix critics and autoregressive policies; advantage over closely related SEAR remains untested.

We agree that SEAR already demonstrates the value of executing a short prefix of a longer action sequence. The specific question HOPS addresses is whether changing how many policy-generated actions are scored during an actor update improves learning when prediction, execution, and critic training remain fixed. On 50 tasks with best-of-32, extending updates from four actions to a uniform mixture of {4, 8, 12, 16} improved the training score by 4.25 percentage points [0.16, 8.24]. We also added a direct 12-task SEAR comparison. Its confidence interval leaves the performance difference between the two systems unresolved (Q4 below).

Unclear design rationale and attribution. Inter-chunk context detachment, uniform horizon sampling, and bootstrap averaging lack sufficient justification. The dynamics of these different choices and their effects is relatively understudied (in prior works as well).

We added separate tests for each decision. Full backpropagation tests the effect of allowing gradients through the generation of later chunks (Appendix C). Fixed-sixteen updates test the choice to sample update lengths (Appendix B.4). A six-condition bootstrap study tests twin-critic reduction across three target-length settings (Appendix B.6). The evidence differs across choices. Sampling reduces computation, mean twin-critic reduction improves performance when averaged across target lengths, and the effect of context detachment on stability remains uncertain.

Questions for Authors

Why detach inter-chunk context? How does full backpropagation affect performance and stability?

We detach the copies of earlier actions fed back into the actor when generating later chunks. This blocks recursive gradient paths through generation while preserving the direct gradient from every critic prefix to each action it contains. For fixed parameters the forward samples are unchanged, and the backward pass computes a semi-gradient.

On 12 tasks with four seeds per task, final success was 80.7% with detachment and 72.0% with full backpropagation (Appendix C, Table 23). This is a descriptive endpoint comparison. For gradient signal-to-noise and agreement across sub-batches, the early- and late-window intervals for the effect of detachment all included zero. We therefore cannot conclude that full backpropagation systematically destabilizes training.

Horizon sampling: Why sample uniformly rather than always use 16 actions? What is the uncertainty in the gain over a maximum horizon of 4?

Uniform sampling generates ten actions per update on average while retaining some sixteen-action updates. At 5M interactions, it took approximately 15 hours on an A100. Always using sixteen added about one hour. In the separate 12-task performance comparison near 2M interactions, fixed-sixteen and sampled updates reached 80.2% and 80.0% final success, respectively (+0.2 points [−13.6, 18.4], Appendix B.4). The lower computation cost motivates the sampled default.

Against fixed-four updates, the 50-task comparison with best-of-32 found a +4.25-point training-score gain [0.16, 8.24]. The final-success difference was +1.44 points [−3.34, 6.67] (Appendix B.2).

Bootstrapping: What motivates averaging bootstrap values across prefix lengths and critics, and choosing H_target = 4? How do these choices affect value estimation and performance?

At each bootstrap state, we sample one four-action continuation and average both target critics’ estimates over its four prefixes. This incorporates several short continuation lengths into the target. The actor objective separately uses the twin-critic minimum at each prefix.

Appendix B.6 crosses mean versus minimum critic reduction with fixed-four, random one-to-four, and fixed-one target lengths. Averaged across these lengths, mean reduction improved final success by 31.5 points [11.7, 46.7] and the training score by 20.7 points [8.5, 32.7]. Under mean reduction, final success was 80.0% for fixed four, 83.5% for random one-to-four, and 76.8% for fixed one. The intervals for the target-length contrasts included zero. We retain four as a short-continuation default. Its optimality and any improvement in value calibration remain open.

Attribution: How does HOPS compare with an MLP actor and SEAR under matched horizons, candidate selection, and training settings?

For the MLP control, we extended the original T-SAC actor’s output and added best-of-32 (Appendix B.3, Table 8). HOPS achieved 80.0% final success and a 66.5% training score, compared with 51.5% and 42.5% for the MLP control. The differences were +28.5 endpoint points [3.3, 49.5] and +23.9 training points [4.4, 40.2]. The Holm-adjusted endpoint sign-flip and Wilcoxon tests gave p=0.108p=0.108 and p=0.161p=0.161. The MLP control retains the T-SAC training setup, which limits attribution to the actor architecture alone.

For SEAR, we matched the 12 Meta-World v3 tasks and the interaction budget of approximately 2.05M (Appendix B.10). HOPS reached 82.8% final success and SEAR-10 executing four actions reached 79.6% (+3.2 points [−17.2, 18.3]). HOPS used best-of-32 and 160 evaluation episodes. SEAR used 20 episodes per execution length, and four was selected after inspecting its execution sweep. These protocol differences and the wide interval leave superiority unresolved.

Could you provide a controlled study linking prefix averaging, horizon sampling, and context detachment to optimization behavior? For example, how do these choices affect gradient variance, critic overestimation, and training stability, and do these effects explain the performance gains?

Appendix C adds a 12-task, four-seed study holding interaction, selection, bootstrap settings, and critic training fixed. During 0.5–1.2M interactions, using only executed-prefix terms lowered log⁡10\log_{10} gradient signal-to-noise by 0.119 relative to HOPS (Holm-adjusted p=0.002p=0.002) and lowered gradient cosine agreement across sub-batches by 0.079 (p=0.005p=0.005). Fixed-four updates lowered the same metrics by 0.111 (p=0.027p=0.027) and 0.083 (p=0.006p=0.006). These differences were inconclusive later in training, and detachment effects remained uncertain in both windows.

Early signed critic bias on executed chunks was higher by 5.76 for executed-prefix-only updates and by 2.06 for fixed-four updates, measured as a percentage of mean absolute observed return. This diagnostic leaves absolute error and calibration on unexecuted tails open. Each configuration also visits its own data and optimizes its own objective. The study identifies an early difference in optimization behavior without establishing that this difference causes improved control.

What motivates the entropy-free default?

We tested actor entropy and soft critic targets separately on 12 tasks with best-of-32 (Appendix B.7). Final success was 80.0% with neither term, 78.3% with actor entropy only, 0.0% with a soft target only, and 75.0% with both. Relative to the default, the actor-only difference was −1.7 points [−14.7, 18.1] and the both-terms difference was −5.0 points [−19.7, 12.8]. The target-only variant, which used automatic temperature tuning, performed substantially worse. We retained the simpler entropy-free setting because adding entropy gave no clear gain in these experiments. The actor remains stochastic at α=0\alpha=0.

Thank you for asking us to clarify how HOPS relates to prior work and which design choices our ablations support.

Actor-update length. We withdraw our earlier claim that extending the actor-update horizon improves learning. We now present the choice of update length as an open research question. In our default Meta-World configuration, we sample the update length uniformly from {4, 8, 12, 16}, so the actor generates 10 actions per update on average.

Reviewer_iLxvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces HOPS, an off-policy actor–critic method that increases the policy-update horizon while also keeping short prediction and execution horizons during interaction. In HOPS, a transformer actor autoregressively generates action chunks, and prefix-conditioned critics evaluate the resulting sequences during actor optimization. The authors evaluate HOPS on all 50 Meta-World ML1 tasks and dense-reward Box Pushing. They report improvements over T-SAC and other baselines. The paper also includes a controlled study of the policy-update horizon and ablations of architectural and training choices.

Critical strengths
  • Conceptually separating the policy-update horizon from the interaction horizons is useful/intuitive, since it allows longer action sequences to be optimized without longer open-loop execution.

  • The empirical results are relatively strong. They cover all 50 Meta-World tasks and there are aggregate improvements over T-SAC.

Critical weaknesses
  • HOPS changes quite a few things relative to T-SAC beyond the actor-update horizon like best-of-32 selection, replay settings, entropy, and target updates. Importantly, removing best-of-N drops the four-task score considerably, so it seems selection is doing a lot of work on those tasks. There is a controlled horizon ablation, which improves the score but Figure 4(a) gives intervals for each configuration rather than the difference, so it's hard to tell how reliable this gain is or how much it explains the main results. More generally, there are many experiments, but not the one the central claim requires. Specifically, the horizon ablation starts at a maximum update horizon of four, so the step from a one-action actor update (T-SAC’s setting) to longer updates is never tested within HOPS.

  • I couldn't find the Box Pushing values for CC, NN, or the interaction horizons. Appendix I also defines the aggregate metric without explaining how Table 1's confidence intervals are computed. There seems to be a mismatch in the collection settings as well. Table 4 gives 4×1004\times100 samples, whereas Figure 4(g) labels HOPS as using eight environments and 800 steps per iteration. Since their own ablation shows that these settings affect performance, it's important to know which configuration produced the main results.

Questions for authors
  1. How were the confidence intervals in Table 1 and Figure 4(a) computed? Specifically, were seeds, tasks, or both resampled, and were the imported baseline intervals constructed using the same procedure? Can you also report uncertainty in the 0.050 improvement from extending the maximum update horizon from four to sixteen using your existing runs?

  2. Can you compare maximum update horizons of four and sixteen with and without best-of-32 selection on a small task subset (keeping the remaining settings fixed)? This would help understand whether the benefit of longer actor updates depends on candidate selection.

  3. Which collection configuration was used for the main results and the horizon ablation?

  4. Can you provide the complete Box Pushing configuration?

  5. How was C=4C=4 selected for Meta-World? Was it used for all 50 tasks in the main comparison?

4 comments

Thank you for reviewing the experimental details so carefully. Your request for uncertainty on the difference itself led us back to the original runs. Your observations about the collection schedule and Box Pushing configuration also helped us correct the reporting and make the experiments easier to reproduce.

Critical Weaknesses

HOPS changes quite a few things relative to T-SAC beyond the actor-update horizon like best-of-32 selection, replay settings, entropy, and target updates. Importantly, removing best-of-N drops the four-task score considerably, so it seems selection is doing a lot of work on those tasks. There is a controlled horizon ablation, which improves the score but Figure 4(a) gives intervals for each configuration rather than the difference, so it's hard to tell how reliable this gain is or how much it explains the main results. More generally, there are many experiments, but not the one the central claim requires. Specifically, the horizon ablation starts at a maximum update horizon of four, so the step from a one-action actor update (T-SAC’s setting) to longer updates is never tested within HOPS.

We added the one-action HOPS control on 12 tasks. With generation, execution, and bootstrap length fixed at one, extending updates from one action to a uniform length from 1 through 16 changed final success by +5.6 percentage points with best-of-32 [−7.4, 22.4] and −2.3 points with a single sample [−21.9, 16.1] (Appendix B.3). The endpoint benefit in this setting remains uncertain.

We also recomputed the original 20-task difference from the underlying runs and added a controlled horizon-by-selection study across all 50 tasks. Figure 4(c) now shows the difference directly. The clearest result is a +4.25-point training-period gain [0.16, 8.24] from extending fixed-four updates under best-of-32. The comparison with T-SAC remains a comparison of the complete systems.

I couldn't find the Box Pushing values for CC, NN, or the interaction horizons. Appendix I also defines the aggregate metric without explaining how Table 1's confidence intervals are computed. There seems to be a mismatch in the collection settings as well. Table 4 gives 4×44\times4 samples, whereas Figure 4(g) labels HOPS as using eight environments and 800 steps per iteration. Since their own ablation shows that these settings affect performance, it's important to know which configuration produced the main results.

The main Meta-World experiments used eight environments collecting 100 transitions each. Table 28 corrects the erroneous four-environment entry. Table 29 now gives the separate Box Pushing configuration, including its two-action generation chunk, four-action interaction horizons, width-256 actor, and single-sample selection. We also clarify that the original Table 1 intervals resampled tasks only. The procedures and settings are detailed below.

Questions for Authors

How were the confidence intervals in Table 1 and Figure 4(a) computed? Specifically, were seeds, tasks, or both resampled, and were the imported baseline intervals constructed using the same procedure? Can you also report uncertainty in the 0.050 improvement from extending the maximum update horizon from four to sixteen using your existing runs?

The original Table 1 and Figure 4(a) used a task-only bootstrap. At each evaluation point, we held each task’s score fixed, resampled tasks with replacement 5,000 times, and recomputed the unweighted mean across tasks. Where seed-level data were available, each task’s score was its interquartile mean across seeds. Imported baselines used the same task-bootstrap procedure. For SAC and PPO, only per-task curves were available.

At 2M interactions, the displayed difference was 0.838−0.788=0.0500.838-0.788=0.050. The unrounded plotted difference was 0.0505. Recomputing the endpoint contrast from the underlying runs gave +4.9 percentage points. With 20,000 resamples per scheme, the 95% intervals were [−0.3, 10.7] for tasks only, [−4.7, 12.8] for seeds within tasks only, and [−6.4, 14.3] for both tasks and seeds. All three intervals include zero.

Can you compare maximum update horizons of four and sixteen with and without best-of-32 selection on a small task subset (keeping the remaining settings fixed)? This would help understand whether the benefit of longer actor updates depends on candidate selection.

We ran this comparison on all 50 Meta-World tasks (Appendix B.2, Tables 6–7). The generation chunk, prediction, execution, and bootstrap horizons were fixed at four actions, and critic training was held fixed. The longer condition sampled actor-update lengths uniformly from {4, 8, 12, 16}.

Longer minus fixed-four updatesTraining score, pp (95% CI)Final success, pp (95% CI)
Best-of-32+4.25 [0.16, 8.24]+1.44 [−3.34, 6.67]
Single sample−0.10 [−5.3, 5.6]−0.6 [−7.7, 6.9]
Difference between selection settings+4.4 [−3.1, 10.9]+2.0 [−6.8, 10.8]

The training-period gain is supported within the default best-of-32 setting. The interaction intervals leave uncertainty about whether the gain depends on selection.

Which collection configuration was used for the main results and the horizon ablation?

Both used 800 transitions per iteration, collected from eight environments with 100 transitions each. Each iteration included four rounds of 100 critic and 40 actor updates. This schedule applies to Table 1 and to the 20-task and 50-task horizon studies. The separate collection-volume ablation intentionally reduced collection to 400 transitions while keeping the optimizer schedule fixed.

Can you provide the complete Box Pushing configuration?

Table 29 gives the configuration for dense-reward FANCY GYM Box Pushing.

SettingValue
Generation chunk2 actions
Prediction, execution, and bootstrap horizons4 actions each
Actor-update lengthUniform over {2, 4, 6, 8, 10, 12, 14, 16}
Candidate selectionSingle sample
Collection4 environments × 100 transitions
Optimizer and learning ratesAdamW, 3×10−53\times10^{-5} for actor and critic
Optimization epochs20 actor, 100 critic
Batch size and replay capacity256 samples, 2,000,000 transitions
Discount and entropy coefficientγ=0.98\gamma=0.98, α=0\alpha=0
Target updatesHard copies, Polyak weight 1.0
Observation and reward normalizationBoth disabled
Transformer depth4 actor layers, 2 critic layers
Actor embedding and attentionWidth 256, 4 heads, 64 dimensions per head
Policy standard deviationState-dependent, predicted by the shared Transformer head
Interaction budget8 million environment interactions

The task uses a 7-DoF Panda arm with seven joint-torque actions. Observations contain joint positions and velocities, box pose, and target pose. Episodes run for 100 steps without early termination. Success at the final step requires position error at most 0.05 m and orientation error at most 0.5 rad.

How was C=4C = 4 selected for Meta-World? Was it used for all 50 tasks in the main comparison?

Yes. We used the same generation chunk length, C=4C=4, for all 50 tasks. In the earlier six-task, four-seed sweep over C∈{1,2,3,4}C\in\{1,2,3,4\}, four had the highest observed aggregate final success at 2M interactions (Figure 8(b)). This choice used benchmark tasks. We did not use the newer 12-task study for selection or retune the chunk length per task.

Thanks for the response. My concerns regarding the collection settings, Box Pushing, and calculation of the Table 1 intervals are clarified.

I do, however, have remaining concerns. Specifically, it is still unclear how much the update horizon contributes to the improvement over T-SAC. The recomputed 20-task gain has intervals that include zero under every resampling scheme. In the 50-task case the final-success difference is +1.44 [−3.34, 6.67] when using best-of-32 and basically zero without it. The training score does improve under best-of-32 (+4.25 [0.16, 8.24]), but that is the only interval that excludes zero. Also, the one-action control appears to be similarly inconclusive.

Lastly, the main text is almost entirely rewritten (besides the preliminaries). Many pages have also been added to the appendix. And, importantly, the evidence for the main claim is from experiments added during the discussion period. This makes it hard to fully evaluate in the little remaining discussion time.

I will keep my score at a 2.

Thank you for asking us to quantify the uncertainty in the difference between configurations. This changed our interpretation of the original 20-task result. When we recomputed the difference in final success between maximum update lengths of 16 and 4 from the underlying runs, the estimate was +4.9 percentage points, with a 95% confidence interval of [−6.4, 14.3] when resampling both tasks and seeds. The intervals from all three resampling schemes included zero. The 50-task final-success comparisons and the one-action controls also leave the effect of update length unresolved. We therefore withdraw our earlier claim that extending the policy-update horizon improves learning. The available evidence does not tell us how much of the HOPS–T-SAC performance gap is attributable to update length.

We also appreciate your point about the amount of new material we asked you to assess late in the discussion. We have restructured the manuscript around the integration of a chunk-autoregressive Transformer actor with T-SAC, using the original submission as our starting point. We retained the original wording wherever possible, substantially reducing the amount of text marked in blue in the main paper. We hope this makes the revisions easier to identify and assess.

Reviewer_UeHQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HOPS, an online off-policy actor-critic method that keeps the prefix-conditioned Transformer critics of T-SAC and replaces the one-step MLP actor with a small causal Transformer that generates action chunks autoregressively. The central claim is that the number of actions the actor is optimized over during an update does not need to match the number of actions executed before replanning, and that extending the former while keeping the latter short improves learning. In the default setting the agent executes 4 actions before replanning, while the actor is trained on sequences of up to 16 actions by averaging the critic values of all prefixes. At interaction time, the executed chunk is picked by best-of-N selection over 32 policy samples, following Q-chunking. HOPS is evaluated on all 50 Meta-World ML1 tasks and on Box Pushing, where it outperforms T-SAC and a range of other baselines.

Critical strengths
  1. The horizon taxonomy is useful. Distinguishing the prediction, execution, and update horizons separates things that are often conflated in the action-chunking literature. The motivation is easy to follow: optimizing longer sequences should help with temporally coordinated behavior, while short execution keeps the agent reactive.

  2. The method is simple. The actor is small and plugs into an existing critic without extra models or offline data, which makes it easy to adopt.

  3. The evaluation is broad and the results are strong. Covering all 50 Meta-World tasks with a strict final-step success criterion, plus Box Pushing, is more thorough than many papers in this area, and the aggregate gains are clear.

  4. The claims are carefully scoped. The authors report non-monotonic and task-dependent ablation results openly and avoid overstating what the system-level comparison shows.

Critical weaknesses
  1. The evidence for the central claim is thin. HOPS differs from T-SAC in many ways beyond the update horizon, including best-of-N selection, the replay buffer, target updates, and entropy, and the ablations show several of these matter more. The only direct evidence is the horizon ablation, where the gain is small, the intervals overlap, task-level effects are not monotonic, and best-of-N stays on. Most importantly, the key control is missing: extending the update and execution horizons together. Without it, one cannot tell whether the benefit comes from decoupling or simply from optimizing longer sequences.

  2. The roles of the actor and best-of-N selection are unclear. HOPS improves the policy twice, once by gradient and again by picking the best of 32 samples, and removing selection more than halves the score. This suggests selection does much of the work, possibly by correcting a mismatch between the actor's sixteen-step objective and the four-step value used for selection. The actor also queries long-prefix critic values on open-loop tails that never appear in replay, and how reliable those values are is not discussed.

  3. The mechanism behind the benefit is not explained. It is unclear why scoring the executed chunk with longer-prefix critics should help. The candidates are lookahead, an auxiliary signal for the shared representation, or variance reduction, and the paper does not separate them.

  4. The ablations are small and tuned on the evaluation tasks. Most use four seeds on four tasks, two of which do not discriminate between configurations, and all design choices were selected on the same benchmark. The default collection setting is also inconsistent between Table 4 and Fig. 13.

  5. The detached actor update is not discussed. Because context actions are detached, the update is a semi-gradient of the stated objective, and the uniform prefix average gives earlier actions more weight without justification.

Questions for authors
  1. Is decoupling what helps? Could you compare the current setting with one where the update and execution horizons are extended together? It would also help to separate fixed from randomly mixed update lengths.

  2. Does the horizon effect hold without best-of-N? Running the horizon ablation with and without selection would show whether selection is correcting a mismatch between the long-horizon actor objective and the executed chunk. What happens if candidates are ranked by the same prefix average that the actor optimizes?

  3. Why are both the gradient-trained actor and best-of-N needed? How does the policy's standard deviation evolve without entropy, and how much does selection contribute as training progresses?

  4. What explains the benefit of longer-prefix credit for the executed chunk? Is it lookahead, an auxiliary representation signal, or variance reduction? One way to probe this is to keep the long update horizon but detach the tail actions from the critic.

  5. Can you isolate the horizon effect against T-SAC? For example, you could run HOPS with a single-action update horizon, and T-SAC with the same best-of-N, replay buffer, and target update.

  6. Which collection setting is the default? Table 4 and Fig. 13 disagree, so please clarify which one produced Table 1 and the ablations.

6 comments

Thank you for the time and care you put into this review. Your distinction between the benefit of longer updates and the benefit of decoupling update and execution horizons helped us frame the central experiment more precisely. Your suggestions about candidate ranking and tail gradients also improved our explanation of what the experiments can establish.

Critical Weaknesses

The evidence for the central claim is thin. HOPS differs from T-SAC in many ways beyond the update horizon, including best-of-N selection, the replay buffer, target updates, and entropy, and the ablations show several of these matter more. The only direct evidence is the horizon ablation, where the gain is small, the intervals overlap, task-level effects are not monotonic, and best-of-N stays on. Most importantly, the key control is missing: extending the update and execution horizons together. Without it, one cannot tell whether the benefit comes from decoupling or simply from optimizing longer sequences.

We agree that the HOPS–T-SAC comparison cannot isolate the horizon effect. We therefore crossed actor-update length with best-of-32 selection on all 50 tasks while holding interaction horizons and critic training fixed. Under best-of-32, longer updates improved the training score by 4.25 percentage points (95% CI [0.16, 8.24]). The endpoint difference was +1.44 points [−3.34, 6.67] (Appendix B.2). We also tested sixteen-action execution on 12 tasks. Its point estimates favored shorter execution, but the intervals were wide (Q1 below). We have narrowed the claim to the training-period benefit observed when interaction remains fixed.

The roles of the actor and best-of-N selection are unclear. HOPS improves the policy twice, once by gradient and again by picking the best of 32 samples, and removing selection more than halves the score. This suggests selection does much of the work, possibly by correcting a mismatch between the actor's sixteen-step objective and the four-step value used for selection. The actor also queries long-prefix critic values on open-loop tails that never appear in replay, and how reliable those values are is not discussed.

The actor learns a distribution over action chunks. Best-of-32 selects among them during collection and evaluation. When evaluation always used a single sample, enabling selection during collection still improved the training score by 18.1 points [0.5, 30.8] (Appendix B.8). This identifies a contribution from data collection. Whether selection also corrects an objective mismatch remains open. We also agree that long generated tails can have limited joint support in replay. Our critic-bias diagnostic covers executed four-action chunks, so calibration on unexecuted tails remains an explicit limitation.

The mechanism behind the benefit is not explained. It is unclear why scoring the executed chunk with longer-prefix critics should help. The candidates are lookahead, an auxiliary signal for the shared representation, or variance reduction, and the paper does not separate them.

A longer-prefix value lets the gradient of an early action depend on a proposed continuation. We tested two consequences of this observation. First, full-prefix optimization gave higher early gradient signal-to-noise and greater agreement across sub-batches than executed-prefix-only optimization (Appendix C). Second, we retained the continuation-dependent values while removing direct gradients to the tail actions (Appendix B.9). The latter comparison was inconclusive. These experiments clarify which gradient paths exist and when the measured gradients differ, while leaving the relative contributions of lookahead, representation learning, and variance reduction unresolved.

The ablations are small and tuned on the evaluation tasks. Most use four seeds on four tasks, two of which do not discriminate between configurations, and all design choices were selected on the same benchmark. The default collection setting is also inconsistent between Table 4 and Fig. 13.

We expanded the central horizon-by-selection study to all 50 tasks. The other new performance studies use a common 12-task subset, generally with four runs per task. Appendix B.1 discloses the three-run endpoint exceptions. We selected the shared chunk length of four using an earlier six-task, four-seed sweep on the benchmark itself, so held-out generalization of that choice remains untested. Table 28 also corrects the collection default to eight environments and 800 transitions per iteration.

The detached actor update is not discussed. Because context actions are detached, the update is a semi-gradient of the stated objective, and the uniform prefix average gives earlier actions more weight without justification.

We now describe the implemented update as a context-detached semi-gradient. Only the copies of earlier actions used to condition later chunks are detached. Each original action remains a differentiable input to every critic prefix containing it. With update length HH, action kk therefore appears in H−k+1H-k+1 objective terms. Earlier actions receive contributions from more terms, although those terms can have different gradient magnitudes.

Appendix C compares full-prefix, final-prefix-only, executed-prefix-only, and full-backpropagation updates. Their descriptive final-success estimates were 80.7%, 76.7%, 76.8%, and 72.0%, respectively. These comparisons document the consequences of alternative choices while leaving the optimal prefix weighting open.

Questions for Authors

Is decoupling what helps? Could you compare the current setting with one where the update and execution horizons are extended together? It would also help to separate fixed from randomly mixed update lengths.

We added both comparisons (Appendix B.4, Tables 9–10). With updates sampled from {4, 8, 12, 16}, executing four actions gave 80.0% final success and a 66.5% training score. Executing sixteen gave 70.6% and 56.7%, respectively. Sixteen minus four was −9.4 endpoint points [−26.9, 13.3] and −9.7 training points [−22.2, 3.2]. Longer execution also requires a longer interaction-time proposal, and the intervals leave the advantage of decoupling unresolved.

With execution fixed at four, always updating over sixteen actions gave 80.2% final success versus 80.0% for mixed updates (+0.2 points [−13.6, 18.4]). Mixed updates generate ten actions on average, which motivates their use as the lower-cost default.

Does the horizon effect hold without best-of-N? Running the horizon ablation with and without selection would show whether selection is correcting a mismatch between the long-horizon actor objective and the executed chunk. What happens if candidates are ranked by the same prefix average that the actor optimizes?

Across all 50 tasks, longer updates changed the training score by +4.25 points with best-of-32 [0.16, 8.24] and −0.10 points with a single sample [−5.3, 5.6]. The interaction was +4.4 points [−3.1, 10.9]. Thus, we find a training-period benefit under best-of-32, but the interaction remains too uncertain to conclude that selection is required.

The current selector ranks the complete four-action candidate by the mean of the two critics. The actor averages the twin-critic minimum over prefixes. We have not evaluated ranking candidates by that same prefix average, so the effect of matching the ranking rule remains open.

Why are both the gradient-trained actor and best-of-N needed? How does the policy's standard deviation evolve without entropy, and how much does selection contribute as training progresses?

All four variants in Appendix B.8 retain the trained actor, so they test where selection helps and leave the necessity of actor training open. With single-sample evaluation, enabling best-of-32 during collection raised the five-checkpoint training score from 42.5% to 60.6% (+18.1 points [0.5, 30.8]). At the final checkpoint the difference was +12.7 points [−9.2, 30.0]. Measuring the immediate benefit of reranking at each stage would require paired evaluations of the same checkpoints.

At zero entropy coefficient, the actor still predicts a state-dependent standard deviation. Measured policy entropy fell sharply early in training and then declined more gradually (Appendix C.5, Figure 31(d)). We did not record trajectories for the individual standard deviations.

What explains the benefit of longer-prefix credit for the executed chunk? Is it lookahead, an auxiliary representation signal, or variance reduction? One way to probe this is to keep the long update horizon but detach the tail actions from the critic.

We ran the suggested intervention (Appendix B.9). We retained all long-prefix critic values and detached critic-input actions after action four. This preserves continuation-dependent gradients for the first four actions while removing direct critic gradients to the tail. Tail detachment changed final success by −8.7 points [−26.0, 11.2] and the training score by −10.5 points [−24.2, 6.3]. Both estimates favor retaining the direct tail gradients, but both intervals include zero. The intervention therefore leaves the causal mechanism unresolved.

Can you isolate the horizon effect against T-SAC? For example, you could run HOPS with a single-action update horizon, and T-SAC with the same best-of-N, replay buffer, and target update.

We added the single-action HOPS control (Appendix B.3, Table 8). With generation, execution, and bootstrap length fixed at one, extending updates from one action to a uniform length from 1 through 16 changed final success from 56.0% to 61.6% with best-of-32 (+5.6 points [−7.4, 22.4]) and from 54.2% to 51.9% with a single sample (−2.3 points [−21.9, 16.1]).

In a separate T-SAC control with one-action execution and no selection, longer updates changed final success from 53.8% to 61.2% (+7.4 points [−15.8, 23.3]). We also tested a T-SAC MLP with longer action outputs and best-of-32. These controls still leave differences in replay and target updates, so we cannot attribute the full HOPS–T-SAC gap to the update horizon.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers and AC for the time and care they devoted to evaluating our manuscript. Their valuable feedback improved the quality of the paper, particularly by identifying the controls needed to distinguish actor-update length from the other components of HOPS. The strongest new evidence is a controlled 50-task study: with interaction horizons and critic-training settings fixed, extending actor updates improves training-period performance under the default best-of-32 (BoN) configuration.

Controlled result. The study compares Hupd=4H_{\mathrm{upd}}=4 with Hupd∼Unif⁡{4,8,12,16}H_{\mathrm{upd}}\sim\operatorname{Unif}\{4,8,12,16\}, holding C=Hpred=Hexec=Htarget=4C=H_{\mathrm{pred}}=H_{\mathrm{exec}}=H_{\mathrm{target}}=4. Under BoN, the five-checkpoint training-period score improves by 4.254.25 percentage points (95% task-and-seed bootstrap CI [0.16,8.24][0.16,8.24]). The endpoint difference is +1.44+1.44 points ([−3.34,6.67][-3.34,6.67]). The training score averages five checkpoints within each run, then computes task-wise seed IQMs and averages tasks; it is distinct from the integral of a plotted aggregate curve (B.1–B.2).

Scope and additional controls. HOPS allows longer actor-update sequences while keeping interaction-time prediction and execution fixed. We support this capability with the controlled result above and separate it from complete-system comparisons with T-SAC. The added one-action, execution-length, selection-stage, and design-choice studies address the requested controls (B.3–B.9); several remain uncertain, including the one-action horizon effect. The new SEAR comparison extends baseline coverage, with protocol differences stated explicitly (B.10).

Method and reporting. We clarify the context-detached semi-gradient and provide a separate optimization-diagnostic study (Appendix C). We also specify the shared Meta-World schedule and the separate Box Pushing configuration (Tables 28–29). The statistical replies distinguish the new difference intervals from the original plotting-derived estimates. These results support improved training-period performance in the tested default setting; they leave endpoint gains, broader generality, and the causal mechanism open.

AuthorsOpenReview ↗

Withdraw our earlier claim that extending the policy-update horizon improves learning

We thank the reviewers and the Area Chair for the care they have given this paper. The reviewers asked what evidence would let us attribute better performance specifically to a longer actor update. On reconsidering the controls, we withdraw our earlier claim that extending the policy-update horizon improves learning.

Our main contribution is a chunk-autoregressive Transformer actor for T-SAC. T-SAC’s critics already estimate the value of every action prefix, while its original actor uses only the first-action value. Our actor generates a chunk of actions in parallel, then conditions the next chunk on the initial state and the actions generated so far. To train the actor, we take the lower of the twin critic estimates at each prefix and average across prefixes. The revised manuscript is titled A Chunk-Autoregressive Actor for T-SAC. Section 5.2 reports the complete-agent comparisons, including 93.8% versus 84.8% success on Meta-World at 5M interactions and 94.1% versus 63.7% on Box Pushing at 8M. These comparisons do not isolate the contribution of any one component.

We distinguish how many actions the actor proposes, how many it executes before replanning, and how many the critics evaluate in an actor update. Section 5.3 presents the preliminary evidence in the main text. On 50 tasks, with prediction and execution fixed at four actions, mixing update lengths from {4, 8, 12, 16} changed final success by +0.4 percentage points relative to fixed-four updates, averaged across the two selection settings (95% CI [−3.8, 5.0]). With best-of-32, the mixed schedule scored 4.25 points higher on the five-checkpoint training measure introduced during discussion (95% CI [0.16, 8.24]). We report that observation with its setting and uncertainty. The research question we take forward is whether, at a fixed replanning interval and training budget, there are tasks for which evaluating actions beyond the next decision improves control.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.