- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks how often diffusion and flow-matching robot policies fail due to unlucky noise draws. A pass@N probe shows that, for example, π0 solves 80.7% of LIBERO-Long initializations with one draw but 96.1% within eight. To recover this headroom online, the authors propose Recency-weighted Temporal Ensembling (RTE), which averages overlapping action chunks with weights that decay exponentially with prediction age, supported by a bias-variance analysis. Across four backbones on LIBERO, SIMPLER, RoboTwin 2.0, and five real-world tasks, RTE mostly beats freshest-only execution at the same stride, especially for long chunks.
Critical strengths
- RTE is simple, training-free, and cheap: the blend takes about 21 to 39 µs per control step against roughly 72 ms per policy call and adds no queries at a given stride. For long chunks, the gains are large: on π0 LIBERO-Long, RTE raises success from 80.7% to 90.1% at the released stride, and on the real robot it beats both open-loop execution and the freshest-only control in all 20 backbone-task combinations. This makes the method easy to adopt in practice.
- The experimental design is broad and well controlled: four backbones, three simulation benchmarks, 2,400 real-world trials, and matched-stride Freshest and Equal controls that separate the effect of replanning more often from that of aggregation. The ablations on multi-sample averaging, low-pass filtering, and shared source noise make the findings about when aggregation helps credible.
Critical weaknesses
- The methodological novelty is limited, and the evidence for its one new ingredient is narrow. As the paper acknowledges (Section 2, Appendix B.3), RTE is ACT's temporal ensembling (Zhao et al., 2023) with the exponential decay reversed to favor newer predictions and defined per control step. The analysis in Section 4.3 is a standard bias-variance argument that the authors themselves relate to classical exponential smoothing (Muth, 1960). The contribution thus rests on showing that recency weighting beats existing rules, but prior weighting rules are only compared on π0 LIBERO-Long (Table 4) and SIMPLER (Table 2). Uniform averaging is actually on par with or ahead of RTE in all six LIBERO settings with H ≤ 16 (Table 1) and in three of six RoboTwin settings (Table 3), but the real-world study has no uniform or ACT-style control. A clear advantage over uniform ensembling is only shown for 50-step chunks, so for short chunks the paper does not establish that RTE is preferable to existing temporal ensembling.
- Several headline claims go beyond what the controls show, and most comparisons lack statistical information. The abstract states that temporal aggregation accounts for most of the gains, but for the 50-step policies on RoboTwin, replanning contributes more than blending in three of four settings (e.g., +16.1 vs. +4.8 points for π0.5 at s=10, Table 3), and the per-task view in Figure 5 (43 of 50 tasks improve) uses s=25, where blending contributes only -0.5 (π0) and +0.6 (π0.5) points. On LIBERO, CogACT's released deployment (94.6%, Table 7) beats RTE at both tested strides. Moreover, neither the RoboTwin results nor the real-world cells (40 trials each) include any statistical information, and LIBERO and SIMPLER only report the std across three evaluation seeds. Many differences in Tables 1 and 2 are within about one std (e.g., RTE vs. Freshest on SIMPLER: 75.1 ± 0.7 vs. 74.2 ± 1.0), and the only confidence interval is the one in Table 10. It is therefore unclear which of the smaller improvements are real and how much of the gain is due to RTE rather than to more frequent replanning.
References:
- T. Z. Zhao, V. Kumar, S. Levine, and C. Finn, "Learning fine-grained bimanual manipulation with low-cost hardware", Robotics: Science and Systems, 2023.
- J. F. Muth, "Optimal properties of exponentially weighted forecasts", Journal of the American Statistical Association, 1960.
Questions for authors
Clarification requests:
- Could the authors add confidence intervals for the RoboTwin and real-world results and paired tests for RTE vs. Equal and RTE vs. Freshest on LIBERO and SIMPLER?
- Why does the real-world study omit the uniform-averaging control? Since uniform weighting is on par with RTE for short chunks in simulation, adding it (at least for CogACT, H=16) would show whether recency weighting itself matters on hardware.
- Base for π0 on LIBERO-Long reaches 80.7%, whereas the openpi repository previously listed 85.2% for its π0 LIBERO recipe with the same replanning stride of 5 (Physical Intelligence, 2025), and the best freshest-only stride comes within 4.1 points of RTE (Figure 4b). Since m was selected on this suite, could the authors explain this gap and report the gain of RTE over a stride-tuned freshest-only control for the other backbones?
- In Table 10, a second Base rollout recovers 61.3% of the failures that pass@4 solves but only 11.5% of the others, so a large gap between the groups is expected for any re-run. Relative to a re-run, RTE in fact gains more on the failures that pass@4 never solves (33 vs. 9 episodes) than on the others (169 vs. 130). How does this support the claim that RTE mainly recovers sampling-induced failures?
- Eq. (4) assumes scatter around a single ideal action, but averaging chunks of a multimodal policy can produce actions between modes. Is this the reason for the performance drops in Appendix C.2 (e.g., π0.5 on Move Stapler Pad at s=10: 64% for Freshest vs. 28% for RTE)?
Suggestion for future work:
- Comparing RTE with selection-based methods such as BID (Liu et al., 2025) at a matched number of policy calls, and combining it with asynchronous execution as in RTC (Black et al., 2025), would show where RTE stands in the compute-performance trade-off and under real-time constraints.
References:
- Physical Intelligence, "openpi", GitHub repository, 2025.
- Y. Liu, J. I. Hamid, A. Xie, Y. Lee, M. Du, and C. Finn, "Bidirectional decoding: Improving action chunking via guided test-time sampling", International Conference on Learning Representations, 2025.
- K. Black, M. Y. Galliker, and S. Levine, "Real-time execution of action chunking flow policies", Advances in Neural Information Processing Systems, 2025.
9 comments
Response to Reviewer f1Fv (2/6)
W2 & Q1: Precision of Headline Claims and Statistical Significance Tests
1. Headline Claim & Case Revisions (in Blue). Thank you for flagging these phrasings and cases; we revised each:
- Abstract, Sec. 1 (Contribution 3), & Sec. 6: Replaced "accounts for most of these gains" globally with matched-stride separation of blending from replanning, noting ESS scaling and orthogonality to test-time methods.
- LIBERO (-Long & CogACT 94.6% Release; Sec. 5.2, 5.4, Fig. 4b, Table 9): Relative to open-loop (66.4%, Table 4), Freshest on -Long rises at (85.9%, 86.0%) before boundary jitter drops it at (80.7%, 77.9%, 73.1%), whereas RTE hits 90.1% at (+9.4% vs. Freshest, ; +4.1% vs. ). On CogACT (, App. B.3, Table 9), frozen RTE () preserves the 94.6 ± 0.2% release at ; Table 1's 93.1% / 92.5% stems from forcing -free CogACT to mid-chunk replanning (), where Freshest drops to 90.9% / 81.9% (65.9% Long) and RTE recovers +2.2% / +10.6% (+21.5% Long).
- RoboTwin 2.0 & Per-Task View (Sec. 5.2 Table 3; App. C.2 Fig. 5, Tables 13–15): Separated 's drift–jitter trade-off from 's monotonic ESS scaling. At (), tail weight preserves Freshest (43.4% vs. 43.7% ; 72.2% vs. 71.9% ) while avoiding Equal's -3.1% to -3.3% drop. App. C.2 Fig. 5 is labeled as total gain (replanning-driven at ), whereas at denser overlap ( on , on GR00T-N1.7; Tables 13–15), isolated blending () improves 39/50, 36/50, and 41/50 tasks (: 49/50 on ).
2. Paired CIs and Significance Tests (App. C.4 Table 23, Tables 1–4, Fig. 3). Table 23 reports 32 paired CIs and two-sided -tests ( suite/task seed blocks, + 20k bootstrap CIs on 4-suite LIBERO & SIMPLER; seed on LIBERO-Long; 50-task on RoboTwin; task on Franka, SE in Fig. 3). Over open-loop Base, RTE gains +23.7% [+18.3, +29.0] on -Long, +21.8% [+17.6, +26.2] and +12.1% [+7.9, +16.1] on RoboTwin (), and +11.8% [+9.2, +14.3] on Franka (; per-backbone Franka vs. Base: +8.0% to +14.5%, all ; vs. Equal: CogACT +7.0% [+4.4, +9.6], +6.0% [+0.8, +11.2]). On SIMPLER (, 2-chunk overlap), RTE beats AAE () and lifts Equal by +2.8% (CI [+0.4, +5.7], , +8.9% on drawer-and-apple), while its +0.9% lead over Freshest (positive on all 3 seeds) has , as do short-horizon vs. Equal pairs at (-1.4% to +0.4%, ; see 1/6):
Table R2: Matched-stride 95% CIs and two-sided paired -tests (from Table 23, App. C.4).
| Benchmark & Backbone () | Comparison | (%) | 95% CI (%) | Paired () |
|---|---|---|---|---|
| LIBERO, () | Freshest / Equal | +3.7 / +5.4% | [+1.9,+5.8] / [+4.2,+6.8] | 3.55 (4.6e-3) / 7.57 (1.1e-5) |
| LIBERO, () | Freshest / Equal | +5.6 / +4.7% | [+3.2,+8.2] / [+3.5,+6.0] | 4.16 (1.6e-3) / 7.08 (2.1e-5) |
| LIBERO, DB-CogACT () | Freshest | +2.2 / +10.6% | [+0.5,+4.1] / [+7.2,+14.4] | 2.26 (4.5e-2) / 5.39 (2.2e-4) |
| LIBERO, () | Freshest | +1.8% | [+0.9,+2.7] | 3.63 (3.9e-3) |
| LIBERO-Long, () | Freshest / Equal | +9.4 / +9.1% | [+7.2,+11.6] / [+6.4,+11.9] | 18.7 (2.9e-3) / 14.4 (4.8e-3) |
| SIMPLER, CogACT-Base () | AAE / Equal | +1.9 / +2.8% | [+0.7,+3.2] / [+0.4,+5.7] | 2.81 (0.017) / 1.92 (0.082) |
| RoboTwin, () | Freshest / Equal | +4.6 / +4.3% | [+1.2,+8.2] / [+2.6,+6.1] | 2.58 (0.013) / 4.77 (1.7e-5) |
| RoboTwin, () | Freshest | +7.0% | [+4.2,+9.9] | 4.78 (1.6e-5) |
| RoboTwin , | Equal | +3.6 / +2.7% | [+2.4,+4.8] / [+1.3,+4.2] | 5.95 (2.8e-7) / 3.72 (5.1e-4) |
| RoboTwin, GR00T-N1.7 () | Freshest | +8.4 / +4.7% | [+5.8,+11.4] / [+3.3,+6.3] | 5.90 (3.4e-7) / 6.19 (1.2e-7) |
| Franka Real, All 4 Backbones | Freshest / Equal | +7.0 / +4.3% | [+5.8,+8.2] / [+2.4,+6.1] | 11.85 (3.2e-10) / 4.77 (1.3e-4) |
Response to Reviewer f1Fv (3/6)
Q3: Why the Baseline Is 80.7% vs. OpenPI's Earlier 85.2%, and Comparison with Stride-Tuned Baselines
1. Why LIBERO-Long Reaches 80.7% at Code Default vs. OpenPI's Earlier 85.2%. We evaluate the unmodified official OpenPI pi0_libero checkpoint (Table 7). While examples/libero/main.py sets a code default of replan_steps = 5, OpenPI did not log whether its earlier 85.2% entry used that default or another CLI flag. Two factors plausibly explain the gap: (i) we conjecture (as a hypothesis, since the exact command was unlogged) that the earlier 85.2% run used , as evaluating the exact same pi0_libero weights under unblended Freshest at yields 86.0 ± 0.9% and 85.9 ± 1.0% over 3 seeds (Fig. 4b), matching 85.2%, whereas discards 45 planned steps every 5 steps and incurs boundary jumps (jump ratio , Table 21); and (ii) at , we evaluate PyTorch-converted weights (bfloat16 SDPA) over 3 seeds 500 episodes rather than JAX/XLA single-run inference. Either way, RTE () at reaches 90.1 ± 0.5%, beating Freshest (+9.4%), best Freshest (+4.1%), and 85.2% (+4.9%). To verify that gains are not specific to pi0_libero or a lower baseline, we fine-tuned a stronger (, 30k steps following MoH [1]; App. C.6, Table 25): concentrating capacity on nearer steps lifts Freshest to 94.8 ± 0.5% (88.1 ± 0.4% Long, above OpenPI's 94.2% / 85.2%), where Equal stalls at 94.8 ± 0.5% (91.1% Long) under 25-step stale tails while frozen RTE reaches 97.3 ± 0.4% (+2.5%) and 94.2 ± 0.3% on Long (+6.1%).
2. Comparing RTE Against Stride-Tuned Freshest Oracles Across All 4 Backbones. Across all 3-seed stride sweeps (Tables 1, 3, 4, 9, 25 & Fig. 4b; plus fixed- SIMPLER in Table 2, 75.1% vs. 74.2% Freshest and 73.2% AAE), frozen RTE () beats or matches the post-hoc stride-tuned Freshest oracle on every backbone (Table R3). On the 3-seed LIBERO-Long grids, () also rises from 93.5% under Freshest (92.5% at ) to 95.2% at (94.5% at ), and GR00T-N1.7 () rises from 92.1% under Freshest to 93.2% at and 93.3% at . In the single-seed (seed 7, 500 episodes/cell) Long scan up to (Fig. 6, ~2%–4% single-run spread at ), likewise tops single-seed Freshest on (90.6% vs. 86.0%), (96.0% at vs. 95.0% at ), and CogACT (92.4% vs. 91.4%), while GR00T-N1.7 has one single-seed Freshest cell of 94.8% at (vs. 92.8% at and 94.0% RTE at ) within single-seed variance:
Table R3: Frozen RTE vs. post-hoc 3-seed stride-tuned Freshest oracle across all 4 backbones.
| Backbone & Benchmark | 3-Seed | Other | Best Freshest | RTE | Gain vs. |
|---|---|---|---|---|---|
| , LIBERO-Long | 1, 3, 5, 10..50 | 80.7% | 86.0±0.9% | 90.1±0.5% | +4.1% |
| -MoH , LIBERO / Long | 5 | — | 94.8 / 88.1% | 97.3 / 94.2% | +2.5 / +6.1% |
| , LIBERO Avg | 3, 5 | 95.8% | 96.8±0.1% | 97.5±0.5% | +0.7% |
| GR00T-N1.7 , LIBERO Avg | 4, 8 | 96.5% | 96.7±0.4% | 97.1±0.6% | +0.4% |
| DB-CogACT , LIBERO Avg | 4, 8, 12–16 | 81.9% | 94.6±0.1% | 94.6±0.2% | +0.0% |
| , RoboTwin 2.0 | 10, 25, 50 | 37.7% | 43.7±0.4% | 44.7±1.0% | +1.0% |
| , RoboTwin 2.0 | 10, 25, 50 | 57.1% | 74.2±0.2% | 78.8±0.1% | +4.6% |
| GR00T-N1.7 , RoboTwin 2.0 | 4, 8, 16 | 26.5% | 33.1±0.7% | 35.2±0.3% | +2.1% |
| Franka Real (All 4, 3,200 trials) | 8, 10, | 75.8% | 80.5% | 87.5% | +7.0% |
Across all 9 backbone–benchmark rows, frozen RTE () matches or beats even a post-hoc stride-tuned Freshest oracle (by up to +6.1% in simulation and +7.0% on physical Franka) without costly per-task stride sweeps.
References:
[1] Jing et al. "Mixture of Horizons in Action Chunking." arXiv, 2025.
Response to Reviewer f1Fv (1/6)
Thank you for your thorough review and for highlighting our matched-stride controls, 4×-compute baseline, multimodal analysis (App. C.2), and physical Franka study. Below and in 2/6–6/6 we add paired significance tests, 800 new Equal hardware trials, and stride-tuned controls (revisions in blue).
W1 & Q2: Methodological Contribution, Short vs. Long Horizons, and Real-World Baselines
Methodological Contribution (Sec. 3–4, Tables 4 & 21; see 7jXp 1/5). Beyond reversing ACT's [1] sign per step, we overturn the smoothness heuristic: ACT [1] has lower jerk than RTE (1.60 vs. 2.95, Table 21) yet drops to 78.9% below s=5 Freshest (80.7%), whereas equicorrelated variance explains why 4× same-state draws stall at 80.3% while cross-observation RTE reaches 90.1% at 1× cost (lifting CogACT's AAE [2] from 81.1% to 88.9%, Table 4).
1. Short-Horizon vs. Long-Horizon Behavior (Sec. 4.3, 5.4, App. A.2, A.5, Table 19). By Proposition 2 (App. A.2), expanding MSE(m) near m=0 (d(Δ)=gΔ) gives m⋆≈nΔˉ(g/σ)2=2sH(H−s)(g/σ)2, at similar per-step g/σ at 20 Hz (App. A.5). Behavior thus tracks drift at maximum chunk age Δmax=(n−1)s (15-row table in 7jXp 2/5, Table R1):
2. Physical Uniform-Averaging Study Across 4 Backbones × 5 Tasks (Sec. 5.3, Fig. 3, Table 23). Following your suggestion, we ran matched-stride Equal (m=0) across all 20 Franka cells (40 shared-layout trials/cell, 3,200 total; testing model-agnostic Equal over CogACT-only AAE [2] and stale-first ACT [1], 78.9%–79.2% in Tables 4 & 21). Pooled success rises from 606/800 (75.8%, Base) and 644/800 (80.5%, Freshest) to 666/800 (83.25%, Equal) and 700/800 (87.50%, RTE; +4.25%, 95% CI [+2.4%, +6.1%], t=4.77, 14 wins, 4 ties, 2 losses in 20 cells):
Table R1: 3,200-trial Franka study (200 trials/backbone/method).
Hardware behavior mirrors Point 1: anchored by 17-D EEF + 7-joint proprioception, GR00T-N1.7 has low drift so Equal nearly matches RTE (83.0% vs. 83.5%), confirming your intuition; conversely, 50-step π0 and π0.5 lag moving Turntable targets under Equal (below Freshest), while qt-free monocular CogACT [2] stalls at 77.5% vs. 84.5% under RTE.
References:
[1] Zhao et al. "Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware." RSS, 2023.
[2] Li et al. "CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation." arXiv, 2024.