- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses action reconstruction and token predictability through two complementary interventions. First, SO(3) augmentation during tokenizer training aims to improve coverage of the action manifold and quantization fidelity. Second, the policy conditions on raw action history and its first- and second-order finite differences. The authors motivate these features through local motion continuity and polynomial extrapolation. Their ablations suggest that explicitly providing the difference features improves success rates beyond conditioning on raw history alone. The method is evaluated on both simulation and real hardware.
Critical strengths
-
The proposed conditioning mechanism is simple and practically useful. Table 1 compares conditioning variants using a shared tokenizer and policy architecture, providing evidence that explicitly exposing finite differences can help policy learning.
-
The experiments investigate several aspects of the method, including action-history conditioning, codebook capacity, chunk size, and tokenizer augmentation. Evaluations on both simulation and real-world tasks, together with experiments adding history conditioning to continuous policies, suggest broader utility.
Critical weaknesses
-
Proposition 4.2 requires strict band-limiting, which does not follow from smooth robot motion alone. Although Appendix A acknowledges this approximation, increasing the retained spectral energy from 99% to 99.9% removes the shrinking-bound guarantee. Without quantifying the omitted frequencies, the bound’s relevance to the original demonstrations remains uncertain. Furthermore, the analysis excludes gripper commands, limiting its ability to explain predictability of the complete action-token sequence.
-
Table 1 does not report uncertainty, and the number of independent training seeds is unclear. Consequently, the robustness of the improvement from raw-history conditioning (62.2%) to history plus finite differences (68.8%) is difficult to assess. Repeated training runs and confidence intervals on the improvement would strengthen this evidence.
-
Section 6.1 reports the best mean success across checkpoints evaluated on test episodes. It is unclear whether checkpoint selection uses separate validation episodes. Selecting checkpoints using the reported test results could produce optimistic performance estimates. An independent final evaluation and clearer reporting of training, tuning, and checkpoint-selection budgets are needed to assess the substantial gains.
Questions for authors
-
How many independent training seeds were used for Table 1? Since these finite differences are deterministic functions of the history, this comparison is particularly relevant to establishing the benefit of their explicit representation. Please report uncertainty across independent training runs and confidence intervals on the improvement.
-
Section 6.1 states that checkpoints are evaluated using 50 test episodes per task and that the best mean success rate is reported. Were checkpoints selected using separate validation episodes, followed by an independent final test evaluation? Please clarify the checkpoint-selection procedure, the number of evaluated checkpoints, and how the reported standard errors were calculated. Given the substantial improvements, comparable training, tuning, and checkpoint-selection budgets across methods are important for interpreting the results.
3 comments
W2 and Q1: Detailed results of Table 1
We thank the reviewer for pointing this.
The uncertainty measure in Table 1: We rerun the controlled study of past-action conditioning on LIBERO-10. Success rates are reported as the mean with standard error. All variants share the same OAT tokenizer (codebook size = 1,000) and policy architecture; only the conditioning inputs differ. Results for (b)–(d) are calculated from four independent training seeds. The OAT baseline retains its previously reported mean and standard error.
| Conditioning Variant | Success Rate (%) ↑ |
|---|---|
| (a) OAT (baseline) | 56.30 ± 1.00 |
| (b) Raw past actions only | 62.25 ± 0.17 |
| (c) Dynamics features only | 63.80 ± 0.14 |
| (d) Dynamics + raw past actions (ours) | 68.70 ± 0.21 |
W3 Q2: Evaluation Protocol and Checkpoint Selection
We thank the reviewer for requesting clarification of our evaluation protocol. We provide the complete procedure below.
-
Training and Checkpoint Selection: For all baselines, we train each policy for 1,000 epochs and evaluate every 200 epochs, resulting in five evaluated checkpoints per training run. At each evaluation, we sample random initial states from the benchmark and evaluate 50 episodes per task. We select the checkpoint with the highest average success rate across tasks within each training run.
-
Independent Seeds and Standard Error: We repeat the entire training and evaluation procedure using four independent random seeds. Each seed produces one selected checkpoint and its corresponding success rate. The final reported result is the mean and standard error across these four selected checkpoints. Therefore, the reported uncertainty reflects variation across independent training runs.
W1: Scope and Validity of the Theoretical Assumptions
We clarify the physical motivation, the scope of Proposition 4.2, and its connection to the complete action-token prediction problem.
Physical Grounding: Under bounded actuator torques and appropriate regularity conditions, robot trajectories are locally smooth (C2), as established in Lemma C.1. This motivates using recent actions to predict local motion trends. Our power spectral density (PSD) analysis on LIBERO-Long demonstrations further shows that 99% of the spectral energy is concentrated below fmax=1.58 Hz, yielding ρ=2πfmaxΔt≈0.50 at Δt=0.05 s (App. A, Table 7). Thus, the band-limited approximation is motivated by the measured frequency distribution of the demonstrated actions, rather than assumed without empirical evidence.
Interpretation of the Contraction Bound: Proposition 4.2 establishes that, under strict band-limiting, the K-point extrapolation error satisfies ‖aₜ − âₜ⁽ᴷ⁾‖∞ ≤ ρᴷ ‖a‖∞. For K = 3 and ρ ≈ 0.50, the bound is approximately 12.5% of the action scale. At the stricter 99.9% spectral-energy cutoff, fₘₐₓ = 4.08 Hz and ρ = 1.28, so the sufficient condition ρ < 1 is no longer satisfied. This means the bound no longer guarantees contraction; it does not imply that past actions cease to provide useful predictive information. Importantly, retaining 99% of spectral energy does not establish a worst-case error guarantee for the original signal, since the remaining high-frequency components may contain meaningful contact transitions and corrections. Our theoretical result therefore characterizes the dominant low-frequency motion component rather than guaranteeing contraction for every demonstrated action..
Applicability to Manipulation and Gripper Commands: Past2Next does not explicitly constrain predicted actions to the extrapolated trajectory. Instead, the finite differences provide local motion information that the policy combines with current visual observations, allowing it to respond to contact transitions and abrupt corrections. The smoothness argument applies to the six continuous pose dimensions, while discrete gripper commands are excluded from Proposition 4.2 and modeled directly by the seven-dimensional action tokenizer. Consequently, the theorem explains a source of predictability in the continuous motion channels. The complete policy's effectiveness is established empirically: past-action conditioning reduces measured next-token predictive entropy from 2.87 to 1.60 nats while improving task success across simulation and real-world manipulation tasks.
Finite-Difference Order and Noise: Lemma D.2 additionally characterizes the trade-off between extrapolation order and noise amplification. Under independent per-step noise with variance σ2, the variance of an m-th order backward difference is (m2m)σ2, giving 2σ2, 6σ2, and 20σ2 for the first three orders. This motivates our use of first- and second-order differences to capture local acceleration and jerk trends without introducing the larger noise amplification of higher-order features.
Overall, our theoretical contribution establishes when local motion history can provide a contracting prediction neighborhood, while our experiments demonstrate that explicitly exposing this structure improves autoregressive action prediction in practical manipulation settings.