- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents a new off-policy AC method denoted HOPS which extends T-SAC. It uses a causal transformer as an actor generating sequences of actions. The sequences are scored by a critic (also a transformer) which ranks and chooses the best actions from the diverse set of sequences. Results show a clear advantage for HOPS over T-SAC on 50 Meta-World ML1 tasks and Box Pushing.
Critical strengths
-
The paper is well-organized and easy to follow.
-
The paper reports strong results across all 50 Meta-World tasks plus Box Pushing
Critical weaknesses
-
The paper doesn't report a one-step actor update () variant of HOPS. As a result, the value from the transformer long update horizon is unclear and unsupported. If the authors can demonstrate the benefits of a longer policy-update horizon over , I will happily consider pushing my score upward.
-
It is unclear whether T-SAC was rerun in the HOPS pipeline or imported from Tian et al. (2026). This is important because HOPS changes several T-SAC settings (that its own ablations show matter: hard target copies, , learning rate, buffer size).
-
Missing baselines for comparison: SEAR (Nagy et al., arXiv March 2026) cited not compared against, GPM (Zhang, Xu and Yu, ICLR 2022).
-
The theoretical analysis is a standard variance identity under untested assumptions, and it does not explain when or why a longer update horizon helps.
Questions for authors
-
Was used for all 50 tasks in Table 1?
-
Is the horizon effect significant across tasks and seeds, for example under a paired or stratified bootstrap?
4 comments
Questions for Authors
Was used for all 50 tasks in Table 1?
Yes. All 50 tasks used , with prediction, execution, and bootstrap horizons fixed at four and actor-update lengths sampled uniformly from {4, 8, 12, 16}. Best-of-32 was used during collection and evaluation. We selected because it had the highest observed aggregate final success at 2M interactions in the earlier six-task, four-seed sweep over (Figure 8(b)). We used the same setting across all 50 tasks without per-task retuning.
Is the horizon effect significant across tasks and seeds, for example under a paired or stratified bootstrap?
The task-and-seed bootstrap supports a training-period improvement under the default best-of-32 setting. Across all 50 tasks, we resampled tasks and then seeds within tasks to estimate the difference between fixed-four updates and updates sampled uniformly from {4, 8, 12, 16} (Appendix B.2, Table 7).
With best-of-32, the five-checkpoint training score improved by 4.25 percentage points [0.16, 8.24]. The final-success difference was +1.44 points [−3.34, 6.67]. With single-sample selection, the corresponding differences were −0.10 [−5.3, 5.6] and −0.6 [−7.7, 6.9]. The training-score interaction was +4.4 points [−3.1, 10.9]. The endpoint effect and the dependence of the training-period gain on selection therefore remain uncertain.
Thank you for the response and the new (honest) results.
W1. I appreciate the one-step control. However, as you state, it does not show a benefit of longer updates over a one-action update. The control also changes the generation chunk to one action, where HOPS reaches 56–62% versus about 80% for the default, so it does not test the main configuration. The strongest remaining evidence is the 50-task training-score gain under best-of-32. The final-success and single-sample differences are not significant, and the training score is a new summary introduced in the rebuttal. Other new results show much larger effects from mean versus min bootstrap reduction (+31.5 points) and from the Transformer actor versus an MLP actor with best-of-32 (+28.5 points). Could it be that HOPS’s advantage comes mainly from these design choices rather than the update horizon?
W2. Thank you for clarifying. Please state this in the table caption.
W3. The SEAR comparison is helpful, though inconclusive. Please also discuss GPM in related work.
W4. Thank you. The early-training gradient diagnostics are a reasonable use of the identity, and the claims are now appropriately scoped.
Overall. The paper’s central claim, stated in its title and abstract, is that extending the policy-update horizon improves learning. After the rebuttal, the evidence supports only a narrow version of this claim: an early-training benefit under best-of-N selection. There's no significant gain in final success and no clear benefit over a one-action update. The paper’s other stated contributions, the horizon framework and the HOPS system with its strong results on all Meta-World tasks and Box Pushing, are well supported. Weighing these, I maintain my score of 3, but it is a low 3. I would give 2.5 if that was possible. I suggest revising the title and abstract so the central claim matches the evidence.
Thank you for suggesting that the title and abstract should reflect what the controls establish. We have followed that suggestion. The revised title is A Chunk-Autoregressive Actor for T-SAC, and we withdraw the claim that longer actor updates improve learning. Your observation about the one-action control is important. It uses a one-action generator, so it leaves open what would happen if we retained the default four-action generator and changed only the update length to one. The five-checkpoint training score was also introduced during rebuttal. We report it as exploratory.
Thank you also for pointing out where readers need the baseline provenance. Appendix J identifies the source. We have added the GPM in the related work.
Thank you for the time you spent evaluating the paper. Your request for a one-step actor control and your questions about baseline provenance and statistical uncertainty helped us test the horizon claim more directly. Your comments also helped us distinguish the empirical findings from the motivation provided by the variance analysis.
Critical Weaknesses
We added the one-step control on 12 Meta-World tasks (Appendix B.3, Table 8). We fixed generation, execution, and bootstrap length at one action and compared a one-action update with a length sampled uniformly from 1 through 16. With best-of-32, final success changed from 56.0% to 61.6% (+5.6 percentage points, 95% CI [−7.4, 22.4]). With single-sample selection, it changed from 54.2% to 51.9% (−2.3 points [−21.9, 16.1]). Both intervals include zero. The endpoint benefit over a one-step update therefore remains uncertain.
Table 1 uses the released T-SAC results from Tian et al. (2026). It compares the complete systems, including their different training and selection settings. Appendix B.3 separately reports new T-SAC controls. With one-action execution and single-sample selection, extending T-SAC’s update changed final success from 53.8% to 61.2% (+7.4 points [−15.8, 23.3]). We also evaluated the original T-SAC MLP with longer outputs and best-of-32. These controls do not fully match the HOPS training pipeline.
We added a SEAR comparison on the same 12 Meta-World v3 tasks at approximately 2.05M interactions (Appendix B.10). HOPS reached 82.8% final success, compared with 79.6% for SEAR-10 executing four actions (+3.2 points [−17.2, 18.3]). HOPS used best-of-32 and 160 evaluation episodes. SEAR used 20 episodes per execution length, and four was its highest observed setting in a sweep from one through ten. The interval and protocol differences leave superiority unresolved. We have no matched GPM result and make no performance claim against it.
We now use the identity to motivate a measurable question about gradient consistency. On 12 tasks, mixed-length updates increased early log10 gradient signal-to-noise by 0.111 relative to fixed-four updates (95% interval [0.027, 0.236], Holm-adjusted p=0.027). Gradient cosine agreement across sub-batches increased by 0.083 [0.037, 0.134] (p=0.006). Both differences were observed during 0.5–1.2M interactions. Neither passed the adjusted threshold later in training (Appendix C, Tables 24–25). These results characterize early optimization under each configuration’s own objective and replay data. Whether they explain improved control remains open.