acceptodds
Desk rejected at ICLR 2027

Temporal Ensembling Revisited: Recovering the Stochastic Headroom of Generative Robot Policies

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion and flow-matching policies formulate robot control as a generative process, decoding each action chunk from sampled noise. Under closed-loop deployment, a failed rollout may therefore reflect an unfavorable sequence of noise draws rather than an inherent limitation of the policy, yet standard success rates leave the two indistinguishable. We examine this ambiguity with an episode-level pass@ probe that holds the task initialization fixed and varies only the policy noise. On LIBERO-Long, success rises from 80.7% with a single draw to 96.1% within eight, indicating that a substantial portion of long-horizon failure is sampling-induced. We recover this stochastic headroom online with Recency-weighted Temporal Ensembling (RTE), which exploits the overlapping predictions inherent to receding-horizon control, averaging them under weights that decay with prediction age. We show analytically that this weighting trades stale-plan bias against prediction noise: when predictions scatter around a plan that drifts with age, neither uniform averaging nor freshest-only execution minimizes per-step action error, and the error-minimizing weights decay with age. With a single fixed decay rate, RTE improves four flow-matching and diffusion backbones in most settings across LIBERO, RoboTwin 2.0, SIMPLER, and a real Franka arm. Controls that replan at matched strides confirm that temporal aggregation accounts for most of these gains, which replanning frequency alone cannot match, and that its contribution tracks the effective ensemble size.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, robot manipulation, action chunking, temporal ensembling, test-time scaling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_f1FvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how often diffusion and flow-matching robot policies fail due to unlucky noise draws. A pass@N probe shows that, for example, π0 solves 80.7% of LIBERO-Long initializations with one draw but 96.1% within eight. To recover this headroom online, the authors propose Recency-weighted Temporal Ensembling (RTE), which averages overlapping action chunks with weights that decay exponentially with prediction age, supported by a bias-variance analysis. Across four backbones on LIBERO, SIMPLER, RoboTwin 2.0, and five real-world tasks, RTE mostly beats freshest-only execution at the same stride, especially for long chunks.

Critical strengths
  • RTE is simple, training-free, and cheap: the blend takes about 21 to 39 µs per control step against roughly 72 ms per policy call and adds no queries at a given stride. For long chunks, the gains are large: on π0 LIBERO-Long, RTE raises success from 80.7% to 90.1% at the released stride, and on the real robot it beats both open-loop execution and the freshest-only control in all 20 backbone-task combinations. This makes the method easy to adopt in practice.
  • The experimental design is broad and well controlled: four backbones, three simulation benchmarks, 2,400 real-world trials, and matched-stride Freshest and Equal controls that separate the effect of replanning more often from that of aggregation. The ablations on multi-sample averaging, low-pass filtering, and shared source noise make the findings about when aggregation helps credible.
Critical weaknesses
  • The methodological novelty is limited, and the evidence for its one new ingredient is narrow. As the paper acknowledges (Section 2, Appendix B.3), RTE is ACT's temporal ensembling (Zhao et al., 2023) with the exponential decay reversed to favor newer predictions and defined per control step. The analysis in Section 4.3 is a standard bias-variance argument that the authors themselves relate to classical exponential smoothing (Muth, 1960). The contribution thus rests on showing that recency weighting beats existing rules, but prior weighting rules are only compared on π0 LIBERO-Long (Table 4) and SIMPLER (Table 2). Uniform averaging is actually on par with or ahead of RTE in all six LIBERO settings with H ≤ 16 (Table 1) and in three of six RoboTwin settings (Table 3), but the real-world study has no uniform or ACT-style control. A clear advantage over uniform ensembling is only shown for 50-step chunks, so for short chunks the paper does not establish that RTE is preferable to existing temporal ensembling.
  • Several headline claims go beyond what the controls show, and most comparisons lack statistical information. The abstract states that temporal aggregation accounts for most of the gains, but for the 50-step policies on RoboTwin, replanning contributes more than blending in three of four settings (e.g., +16.1 vs. +4.8 points for π0.5 at s=10, Table 3), and the per-task view in Figure 5 (43 of 50 tasks improve) uses s=25, where blending contributes only -0.5 (π0) and +0.6 (π0.5) points. On LIBERO, CogACT's released deployment (94.6%, Table 7) beats RTE at both tested strides. Moreover, neither the RoboTwin results nor the real-world cells (40 trials each) include any statistical information, and LIBERO and SIMPLER only report the std across three evaluation seeds. Many differences in Tables 1 and 2 are within about one std (e.g., RTE vs. Freshest on SIMPLER: 75.1 ± 0.7 vs. 74.2 ± 1.0), and the only confidence interval is the one in Table 10. It is therefore unclear which of the smaller improvements are real and how much of the gain is due to RTE rather than to more frequent replanning.

References:

  • T. Z. Zhao, V. Kumar, S. Levine, and C. Finn, "Learning fine-grained bimanual manipulation with low-cost hardware", Robotics: Science and Systems, 2023.
  • J. F. Muth, "Optimal properties of exponentially weighted forecasts", Journal of the American Statistical Association, 1960.
Questions for authors

Clarification requests:

  • Could the authors add confidence intervals for the RoboTwin and real-world results and paired tests for RTE vs. Equal and RTE vs. Freshest on LIBERO and SIMPLER?
  • Why does the real-world study omit the uniform-averaging control? Since uniform weighting is on par with RTE for short chunks in simulation, adding it (at least for CogACT, H=16) would show whether recency weighting itself matters on hardware.
  • Base for π0 on LIBERO-Long reaches 80.7%, whereas the openpi repository previously listed 85.2% for its π0 LIBERO recipe with the same replanning stride of 5 (Physical Intelligence, 2025), and the best freshest-only stride comes within 4.1 points of RTE (Figure 4b). Since m was selected on this suite, could the authors explain this gap and report the gain of RTE over a stride-tuned freshest-only control for the other backbones?
  • In Table 10, a second Base rollout recovers 61.3% of the failures that pass@4 solves but only 11.5% of the others, so a large gap between the groups is expected for any re-run. Relative to a re-run, RTE in fact gains more on the failures that pass@4 never solves (33 vs. 9 episodes) than on the others (169 vs. 130). How does this support the claim that RTE mainly recovers sampling-induced failures?
  • Eq. (4) assumes scatter around a single ideal action, but averaging chunks of a multimodal policy can produce actions between modes. Is this the reason for the performance drops in Appendix C.2 (e.g., π0.5 on Move Stapler Pad at s=10: 64% for Freshest vs. 28% for RTE)?

Suggestion for future work:

  • Comparing RTE with selection-based methods such as BID (Liu et al., 2025) at a matched number of policy calls, and combining it with asynchronous execution as in RTC (Black et al., 2025), would show where RTE stands in the compute-performance trade-off and under real-time constraints.

References:

  • Physical Intelligence, "openpi", GitHub repository, 2025.
  • Y. Liu, J. I. Hamid, A. Xie, Y. Lee, M. Du, and C. Finn, "Bidirectional decoding: Improving action chunking via guided test-time sampling", International Conference on Learning Representations, 2025.
  • K. Black, M. Y. Galliker, and S. Levine, "Real-time execution of action chunking flow policies", Advances in Neural Information Processing Systems, 2025.
9 comments
AuthorsOpenReview ↗

Response to Reviewer f1Fv (1/6)

Thank you for your thorough review and for highlighting our matched-stride controls, 4×4\times-compute baseline, multimodal analysis (App. C.2), and physical Franka study. Below and in 2/6–6/6 we add paired significance tests, 800 new Equal hardware trials, and stride-tuned controls (revisions in blue).

W1 & Q2: Methodological Contribution, Short vs. Long Horizons, and Real-World Baselines

Methodological Contribution (Sec. 3–4, Tables 4 & 21; see 7jXp 1/5). Beyond reversing ACT's [1] sign per step, we overturn the smoothness heuristic: ACT [1] has lower jerk than RTE (1.601.60 vs. 2.952.95, Table 21) yet drops to 78.9% below s=5s = 5 Freshest (80.7%), whereas equicorrelated variance explains why 4×4\times same-state draws stall at 80.3% while cross-observation RTE reaches 90.1% at 1×1\times cost (lifting CogACT's AAE [2] from 81.1% to 88.9%, Table 4).

1. Short-Horizon vs. Long-Horizon Behavior (Sec. 4.3, 5.4, App. A.2, A.5, Table 19). By Proposition 2 (App. A.2), expanding MSE(m)\mathrm{MSE}(m) near m=0m = 0 (d(Δ)=gΔd(\Delta) = g\Delta) gives m⋆≈nΔˉ(g/σ)2=H(H−s)2s(g/σ)2m^\star \approx n\bar{\Delta}(g/\sigma)^2 = \frac{H(H - s)}{2s}(g/\sigma)^2, at similar per-step g/σg/\sigma at 20 Hz (App. A.5). Behavior thus tracks drift at maximum chunk age Δmax⁡=(n−1)s\Delta_{\max} = (n - 1)s (15-row table in 7jXp 2/5, Table R1):

  • Short-chunk low-drift regime (H∈[10,16],s∈[3,8]H \in [10, 16], s \in [3, 8]): With n∈[2,4]n \in [2, 4] chunks and Δmax⁡≤12\Delta_{\max} \le 12 steps (≤0.6\le 0.6 s at 20 Hz; nΔˉ∈[5,24]n\bar{\Delta} \in [5, 24] vs. 225225 at H=50H = 50), drift stays mild relative to noise (η^∈[0.4,1.2]\hat{\eta} \in [0.4, 1.2] vs. 3.93.9 on π0\pi_0; App. A.5), placing m⋆∈(0,0.05]m^\star \in (0, 0.05]. Frozen m=0.1m = 0.1 (chosen on H=50H = 50) beats Equal on 2/8 rows and stays within 0.7%–1.4% (mean 1.0%) on the other 6, while halving to m=0.05m = 0.05 beats m=0.1m = 0.1 on all 7 settings in Table 19 and matches or beats Equal on 5/7 (93.2% vs. 92.7% on CogACT [2]; 98.1%–98.3% on GR00T-N1.7; 97.7% on π0.5\pi_{0.5}).
  • High-drift and long-horizon regimes (s=1s = 1 domain shift and H∈[40,50]H \in [40, 50]): On SIMPLER Visual Matching (H=16,s=1H = 16, s = 1, Table 2), domain shift and per-step replanning raise 15-step drift, so m=0.1m = 0.1 beats AAE [2] (75.1% vs. 73.2%, +1.9%, p=0.017p = 0.017) and lifts Equal from 72.3% (+2.8%, 95% CI [+0.4, +5.7], +8.9% on drawer-and-apple). On H∈[40,50]H \in [40, 50] (Δmax⁡=25\Delta_{\max} = 25–4949 steps, 1.25–2.45 s), tail drift dominates σ\sigma (η^=3.9\hat{\eta} = 3.9, Fig. 1b): Equal puts 50% weight on 25–49-step stale plans, trailing RTE by 8.0%–9.2% on π0\pi_0-Long (Table 4) and falling 3.1%–3.3% below s=25s = 25 Freshest on RoboTwin 2.0 (Table 3). Across Tables 1–3, frozen m=0.1m = 0.1 wins 9/15 rows, trading 0.7%–1.4% on short chunks to avoid 3.1%–9.2% stale-tail drops.

2. Physical Uniform-Averaging Study Across 4 Backbones ×\times 5 Tasks (Sec. 5.3, Fig. 3, Table 23). Following your suggestion, we ran matched-stride Equal (m=0m = 0) across all 20 Franka cells (40 shared-layout trials/cell, 3,200 total; testing model-agnostic Equal over CogACT-only AAE [2] and stale-first ACT [1], 78.9%–79.2% in Tables 4 & 21). Pooled success rises from 606/800 (75.8%, Base) and 644/800 (80.5%, Freshest) to 666/800 (83.25%, Equal) and 700/800 (87.50%, RTE; +4.25%, 95% CI [+2.4%, +6.1%], t=4.77t = 4.77, 14 wins, 4 ties, 2 losses in 20 cells):

Table R1: 3,200-trial Franka study (200 trials/backbone/method).

Backbone (H,s)(H, s)BaseFreshestEqualRTETurntable (Fresh/Equal/RTE)ΔEqual\Delta_{\mathrm{Equal}}
GR00T-N1.7 (40,10)(40, 10)69.0%76.5%83.0%83.5%77.5 / 80.0 / 80.0%+0.5%
π0\pi_0 (50,10)(50, 10)73.5%80.0%81.5%87.5%80.0 / 77.5 / 87.5%+6.0% (p=0.033p = 0.033)
π0.5\pi_{0.5} (50,10)(50, 10)84.0%89.5%91.0%94.5%87.5 / 85.0 / 92.5%+3.5% (p=0.080p = 0.080)
CogACT (16,8)(16, 8)76.5%76.0%77.5%84.5%77.5 / 77.5 / 82.5%+7.0% (p=0.0017p = 0.0017)
All 4 (800/method)75.8%80.5%83.3%87.5%80.6 / 80.0 / 85.6%+4.3% (p=0.00013p = 0.00013)

Hardware behavior mirrors Point 1: anchored by 17-D EEF + 7-joint proprioception, GR00T-N1.7 has low drift so Equal nearly matches RTE (83.0% vs. 83.5%), confirming your intuition; conversely, 50-step π0\pi_0 and π0.5\pi_{0.5} lag moving Turntable targets under Equal (below Freshest), while qtq_t-free monocular CogACT [2] stalls at 77.5% vs. 84.5% under RTE.

References:

[1] Zhao et al. "Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware." RSS, 2023.

[2] Li et al. "CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation." arXiv, 2024.

AuthorsOpenReview ↗

Response to Reviewer f1Fv (2/6)

W2 & Q1: Precision of Headline Claims and Statistical Significance Tests

1. Headline Claim & Case Revisions (in Blue). Thank you for flagging these phrasings and cases; we revised each:

  • Abstract, Sec. 1 (Contribution 3), & Sec. 6: Replaced "accounts for most of these gains" globally with matched-stride separation of blending from replanning, noting ESS scaling and orthogonality to test-time methods.
  • LIBERO (π0\pi_0-Long & CogACT 94.6% Release; Sec. 5.2, 5.4, Fig. 4b, Table 9): Relative to open-loop s=50s = 50 (66.4%, Table 4), Freshest on π0\pi_0-Long rises at s=15,10s = 15, 10 (85.9%, 86.0%) before boundary jitter drops it at s=5,3,1s = 5, 3, 1 (80.7%, 77.9%, 73.1%), whereas RTE hits 90.1% at s=5s = 5 (+9.4% vs. s=5s = 5 Freshest, p=0.0029p = 0.0029; +4.1% vs. s=10s = 10). On CogACT (H=16H = 16, App. B.3, Table 9), frozen RTE (m=0.1m = 0.1) preserves the 94.6 ± 0.2% release at s=12−16s = 12{-}16; Table 1's 93.1% / 92.5% stems from forcing qtq_t-free CogACT to mid-chunk replanning (s=8,4s = 8, 4), where Freshest drops to 90.9% / 81.9% (65.9% Long) and RTE recovers +2.2% / +10.6% (+21.5% Long).
  • RoboTwin 2.0 & Per-Task View (Sec. 5.2 Table 3; App. C.2 Fig. 5, Tables 13–15): Separated Δreplan\Delta_{\mathrm{replan}}'s drift–jitter trade-off from Δblend\Delta_{\mathrm{blend}}'s monotonic ESS scaling. At s=25s = 25 (H=50,ESS=1.2H = 50, \mathrm{ESS} = 1.2), tail weight ≤e−2.5=0.08\le e^{-2.5} = 0.08 preserves Freshest (43.4% vs. 43.7% π0\pi_0; 72.2% vs. 71.9% π0.5\pi_{0.5}) while avoiding Equal's -3.1% to -3.3% drop. App. C.2 Fig. 5 is labeled as total gain Δ=Δreplan+Δblend\Delta = \Delta_{\mathrm{replan}} + \Delta_{\mathrm{blend}} (replanning-driven at s=25s = 25), whereas at denser overlap (s=10s = 10 on π0/π0.5\pi_0 / \pi_{0.5}, s=4s = 4 on GR00T-N1.7; Tables 13–15), isolated blending (RTE−Freshest\mathrm{RTE} - \mathrm{Freshest}) improves 39/50, 36/50, and 41/50 tasks (RTE−Base\mathrm{RTE} - \mathrm{Base}: 49/50 on π0.5\pi_{0.5}).

2. Paired CIs and Significance Tests (App. C.4 Table 23, Tables 1–4, Fig. 3). Table 23 reports 32 paired CIs and two-sided tt-tests (1212 suite/task ×\times seed blocks, df=11\mathrm{df} = 11 + 20k bootstrap CIs on 4-suite LIBERO & SIMPLER; seed df=2\mathrm{df} = 2 on LIBERO-Long; 50-task df=49\mathrm{df} = 49 on RoboTwin; task df=4,19\mathrm{df} = 4, 19 on Franka, ±1\pm 1 SE in Fig. 3). Over open-loop Base, RTE gains +23.7% [+18.3, +29.0] on π0\pi_0-Long, +21.8% [+17.6, +26.2] and +12.1% [+7.9, +16.1] on RoboTwin π0.5/π0\pi_{0.5}/\pi_0 (s=10s = 10), and +11.8% [+9.2, +14.3] on Franka (p≤0.0028p \le 0.0028; per-backbone Franka vs. Base: +8.0% to +14.5%, all p≤0.012p \le 0.012; vs. Equal: CogACT +7.0% [+4.4, +9.6], π0\pi_0 +6.0% [+0.8, +11.2]). On SIMPLER (s=1s = 1, 2-chunk overlap), RTE beats AAE (p=0.017p = 0.017) and lifts Equal by +2.8% (CI [+0.4, +5.7], p=0.082p = 0.082, +8.9% on drawer-and-apple), while its +0.9% lead over Freshest (positive on all 3 seeds) has p>0.10p \gt 0.10, as do short-horizon H≤16H \le 16 vs. Equal pairs at s∈[3,8]s \in [3, 8] (-1.4% to +0.4%, p>0.10p \gt 0.10; see 1/6):

Table R2: Matched-stride 95% CIs and two-sided paired tt-tests (from Table 23, App. C.4).

Benchmark & Backbone (ss)ComparisonΔ\Delta (%)95% CI (%)Paired tt (pp)
LIBERO, π0\pi_0 (s=5s=5)Freshest / Equal+3.7 / +5.4%[+1.9,+5.8] / [+4.2,+6.8]3.55 (4.6e-3) / 7.57 (1.1e-5)
LIBERO, π0\pi_0 (s=3s=3)Freshest / Equal+5.6 / +4.7%[+3.2,+8.2] / [+3.5,+6.0]4.16 (1.6e-3) / 7.08 (2.1e-5)
LIBERO, DB-CogACT (s=8/4s=8/4)Freshest+2.2 / +10.6%[+0.5,+4.1] / [+7.2,+14.4]2.26 (4.5e-2) / 5.39 (2.2e-4)
LIBERO, π0.5\pi_{0.5} (s=3s=3)Freshest+1.8%[+0.9,+2.7]3.63 (3.9e-3)
LIBERO-Long, π0\pi_0 (s=5s=5)Freshest / Equal+9.4 / +9.1%[+7.2,+11.6] / [+6.4,+11.9]18.7 (2.9e-3) / 14.4 (4.8e-3)
SIMPLER, CogACT-Base (s=1s=1)AAE / Equal+1.9 / +2.8%[+0.7,+3.2] / [+0.4,+5.7]2.81 (0.017) / 1.92 (0.082)
RoboTwin, π0.5\pi_{0.5} (s=10s=10)Freshest / Equal+4.6 / +4.3%[+1.2,+8.2] / [+2.6,+6.1]2.58 (0.013) / 4.77 (1.7e-5)
RoboTwin, π0\pi_0 (s=10s=10)Freshest+7.0%[+4.2,+9.9]4.78 (1.6e-5)
RoboTwin s=25s=25, π0.5/π0\pi_{0.5}/\pi_0Equal+3.6 / +2.7%[+2.4,+4.8] / [+1.3,+4.2]5.95 (2.8e-7) / 3.72 (5.1e-4)
RoboTwin, GR00T-N1.7 (s=4/8s=4/8)Freshest+8.4 / +4.7%[+5.8,+11.4] / [+3.3,+6.3]5.90 (3.4e-7) / 6.19 (1.2e-7)
Franka Real, All 4 BackbonesFreshest / Equal+7.0 / +4.3%[+5.8,+8.2] / [+2.4,+6.1]11.85 (3.2e-10) / 4.77 (1.3e-4)
AuthorsOpenReview ↗

Response to Reviewer f1Fv (3/6)

Q3: Why the s=5s=5 Baseline Is 80.7% vs. OpenPI's Earlier 85.2%, and Comparison with Stride-Tuned Baselines

1. Why π0\pi_0 LIBERO-Long Reaches 80.7% at Code Default s=5s = 5 vs. OpenPI's Earlier 85.2%. We evaluate the unmodified official OpenPI pi0_libero checkpoint (Table 7). While examples/libero/main.py sets a code default of replan_steps = 5, OpenPI did not log whether its earlier 85.2% entry used that default or another CLI flag. Two factors plausibly explain the gap: (i) we conjecture (as a hypothesis, since the exact command was unlogged) that the earlier 85.2% run used s=10s = 10, as evaluating the exact same pi0_libero weights under unblended Freshest at s=10,15s = 10, 15 yields 86.0 ± 0.9% and 85.9 ± 1.0% over 3 seeds (Fig. 4b), matching 85.2%, whereas s=5s = 5 discards 45 planned steps every 5 steps and incurs boundary jumps (jump ratio 2.622.62, Table 21); and (ii) at s=5s = 5, we evaluate PyTorch-converted weights (bfloat16 SDPA) over 3 seeds ×\times 500 episodes rather than JAX/XLA single-run inference. Either way, RTE (m=0.1m = 0.1) at s=5s = 5 reaches 90.1 ± 0.5%, beating s=5s = 5 Freshest (+9.4%), best s⋆=10s^\star = 10 Freshest (+4.1%), and 85.2% (+4.9%). To verify that gains are not specific to pi0_libero or a lower baseline, we fine-tuned a stronger π0\pi_0 (H=30H = 30, 30k steps following MoH [1]; App. C.6, Table 25): concentrating capacity on nearer steps lifts s=5s = 5 Freshest to 94.8 ± 0.5% (88.1 ± 0.4% Long, above OpenPI's 94.2% / 85.2%), where Equal stalls at 94.8 ± 0.5% (91.1% Long) under 25-step stale tails while frozen m=0.1m = 0.1 RTE reaches 97.3 ± 0.4% (+2.5%) and 94.2 ± 0.3% on Long (+6.1%).

2. Comparing RTE Against Stride-Tuned Freshest Oracles Across All 4 Backbones. Across all 3-seed stride sweeps (Tables 1, 3, 4, 9, 25 & Fig. 4b; plus fixed-s=1s = 1 SIMPLER in Table 2, 75.1% vs. 74.2% Freshest and 73.2% AAE), frozen RTE (m=0.1m = 0.1) beats or matches the post-hoc stride-tuned Freshest oracle s⋆s^\star on every backbone (Table R3). On the 3-seed LIBERO-Long grids, π0.5\pi_{0.5} (s=1,2,3,5,8s = 1, 2, 3, 5, 8) also rises from 93.5% under s⋆=8s^\star = 8 Freshest (92.5% at s=5s = 5) to 95.2% at s=8s = 8 (94.5% at s=3s = 3), and GR00T-N1.7 (s=1,2,4,8s = 1, 2, 4, 8) rises from 92.1% under s⋆=8s^\star = 8 Freshest to 93.2% at s=8s = 8 and 93.3% at s=1s = 1. In the single-seed (seed 7, 500 episodes/cell) Long scan up to s=Hs = H (Fig. 6, ~2%–4% single-run spread at s=Hs = H), m=0.1m = 0.1 likewise tops single-seed Freshest on π0\pi_0 (90.6% vs. 86.0%), π0.5\pi_{0.5} (96.0% at s=8s = 8 vs. 95.0% at s=10s = 10), and CogACT (92.4% vs. 91.4%), while GR00T-N1.7 has one single-seed Freshest cell of 94.8% at s=12s = 12 (vs. 92.8% at s=16s = 16 and 94.0% RTE at s=1s = 1) within single-seed variance:

Table R3: Frozen RTE (m=0.1)(m=0.1) vs. post-hoc 3-seed stride-tuned Freshest oracle (s⋆)(s^\star) across all 4 backbones.

Backbone & Benchmark3-Seed ssOther ssBest Freshest (s⋆)(s^\star)RTE (m=0.1)(m=0.1)Gain vs. s⋆s^\star
π0\pi_0 (H=50)(H=50), LIBERO-Long1, 3, 5, 10..5080.7% (s=5)(s=5)86.0±0.9% (s⋆=10)(s^\star=10)90.1±0.5% (s=5)(s=5)+4.1%
π0\pi_0-MoH (H=30)(H=30), LIBERO / Long5—94.8 / 88.1% (s=5)(s=5)97.3 / 94.2% (s=5)(s=5)+2.5 / +6.1%
π0.5\pi_{0.5} (H=10)(H=10), LIBERO Avg3, 595.8% (s=3)(s=3)96.8±0.1% (s⋆=5)(s^\star=5)97.5±0.5% (s=3)(s=3)+0.7%
GR00T-N1.7 (H=16)(H=16), LIBERO Avg4, 896.5% (s=4)(s=4)96.7±0.4% (s⋆=8)(s^\star=8)97.1±0.6% (s=8)(s=8)+0.4%
DB-CogACT (H=16)(H=16), LIBERO Avg4, 8, 12–1681.9% (s=4)(s=4)94.6±0.1% (s⋆=12−16)(s^\star=12{-}16)94.6±0.2% (s=12−16)(s=12{-}16)+0.0%
π0\pi_0 (H=50)(H=50), RoboTwin 2.010, 25, 5037.7% (s=10)(s=10)43.7±0.4% (s⋆=25)(s^\star=25)44.7±1.0% (s=10)(s=10)+1.0%
π0.5\pi_{0.5} (H=50)(H=50), RoboTwin 2.010, 25, 5057.1% (s=50)(s=50)74.2±0.2% (s⋆=10)(s^\star=10)78.8±0.1% (s=10)(s=10)+4.6%
GR00T-N1.7 (H=16)(H=16), RoboTwin 2.04, 8, 1626.5% (s=4)(s=4)33.1±0.7% (s⋆=16)(s^\star=16)35.2±0.3% (s=8)(s=8)+2.1%
Franka Real (All 4, 3,200 trials)8, 10, HH75.8% (s=H)(s=H)80.5% (s⋆=8,10)(s^\star=8,10)87.5% (s=8,10)(s=8,10)+7.0%

Across all 9 backbone–benchmark rows, frozen RTE (m=0.1m = 0.1) matches or beats even a post-hoc stride-tuned Freshest oracle s⋆s^\star (by up to +6.1% in simulation and +7.0% on physical Franka) without costly per-task stride sweeps.

References:

[1] Jing et al. "Mixture of Horizons in Action Chunking." arXiv, 2025.

Reviewer_WiU4OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Recency-weighted Temporal Ensembling (RTE) to mitigate sampling-induced failures in generative robot policies without requiring extra compute or retraining. RTE temporally aligns the overlapping action chunks naturally produced by receding-horizon control and averages them using an exponential recency decay. The authors theoretically demonstrate that this age-decayed weighting optimally balances single-draw prediction noise against stale-plan trajectory drift. Empirical evaluations across four backbones (CogACT, π0\pi_0, π0.5\pi_{0.5}, GR00T-N1.7) in simulation and on a physical Franka arm show that RTE systematically improves success rates over both freshest-only and uniform-averaging baselines.

Critical strengths
  • The formalization of the bias-variance trade-off in temporal ensembling (Propositions 1 and 2) is a strong contribution. By mathematically demonstrating why both freshest-only (high variance) and uniform averaging (high stale-plan bias) are suboptimal limits, the authors provide a rigorous first-principles justification for exponential recency weighting.

  • The method is training-free, parameter-free, and operates exclusively on the client side with negligible overhead (∼21−39μs\sim 21-39 \mu s per step). The ability to reliably boost inference success across multiple architectures and action spaces without fine-tuning makes this an highly actionable, plug-and-play formulation for the robotics community.

  • Authors also provide a broad evaluation across four backbones (π0, π0.5, GR00T-N1.7, CogACT), three simulation benchmarks (LIBERO, SIMPLER, RoboTwin 2.0), and real-world Franka tasks, with matched-stride baselines (Freshest and Equal) and decomposition of gains into Δreplan vs Δblend. Ablations on decay rate, stride, ensembling heuristics (ACT, AAE, VOTE), and effective ensemble size (ESS) confirms the core claims.

  • The paper is well-motivated, easy to follow with intuitive figures.

Critical weaknesses
  • About the novelty: the paper somehow resembles prior temporal aggregation (e.g., ACT’s exponential smoothing). The main difference lies in principled analysis and emphasizes decay by physical age.

  • The current comparison with the test-time scaling method is quite limited. Authors should consider adding a few more advanced techniques, such as RoboMonkey or TACO (mentioned in related works). Also there are other relevant ons such as Denoising-Variance Adaptive Chunking method [1] or [2] have not been discussed.

  • About real-world robot: to run the pass@N diagnostic, it's unclear to the Reviewer how it is implemented. In particular, in a simulator like LIBERO, we can simply reload a saved environment state down to the exact floating-point numbers. In the real world, it is physically impossible to reset a table, objects, and a Franka robot arm to the exact micrometer-level starting conditions for multiple identical trials.

[1] Denoising Tells When to Replan: Denoising-Variance Adaptive Chunking for Flow-Based Robot Policies

[2] Start Right, Arrive Right: Asynchronous Execution via Initial Noise Selection, CoRL 2026

Questions for authors

I have the following questions and suggestions:

  • Uniform averaging sometimes outperforms RTE on short horizons. Can you provide a simple practitioner guideline (in terms of horizon H, stride s, and an estimate of disagreement growth) for choosing between RTE vs. uniform?

  • The decay rate m=0.1m=0.1 was calibrated on π0\pi_0-Long and transferred successfully across all other backbones and tasks. To what extent is this constant dependent on the control frequency of the underlying dataset? If a dataset was collected at 50Hz instead of 10Hz, would mm require scaling?

  • Equation 3 Implementation Details: In real-world deployments where control loops might experience minor asynchronous jitters or frame drops, is physical age Δk\Delta_k tracked via absolute wall-clock timestamps or strict control-step indices?

  • The proposed RTE framework reduces sampling variance through post-hoc temporal blending of independent chunks. Recent concurrent work, such as PAINT [2], addresses a similar chunk-overlap challenge but operates pre-generatio, using flow inversion and noise-level conditioning to enforce smooth asynchronous handoffs without relying on post-hoc averaging. Could the authors discuss the trade-offs between these two philosophies? Specifically, does RTE's reliance on completely independent noise draws limit its ability to achieve the kind of exact boundary-matching seen in inversion-based methods, or could these two approaches be orthogonally combined?

7 comments
AuthorsOpenReview ↗

Response to Reviewer WiU4 (1/6)

Dear Reviewer WiU4,

Thank you for your constructive review and for recognizing Propositions 1–2, RTE's training-free client-side simplicity, and our matched-stride evaluations across 4 backbones and physical Franka hardware. Revisions are highlighted in blue.

W1: Algorithmic Novelty and Relationship to ACT

Thank you for this thoughtful point. As reflected in our title ("Temporal Ensembling Revisited"), RTE intentionally retains a minimal zero-extra-query exponential operator rather than introducing a complex module. Notably, while ACT [1] weights oldest chunks highest (m<0m \lt 0) to smooth motion, revisiting this primitive from first principles changes its objective, weighting direction, and step-age parameterization:

1. Recovering Stochastic Headroom Rather Than Motion Smoothing (Sec. 3 & 5.4, Tables 4, 20 & 21, App. A.4). Prior work typically treated temporal aggregation as a heuristic for smoothing jerky actions, often assuming smoother trajectories drive higher success. Our smoothness ablation on π0\pi_0 LIBERO-Long (s=5s = 5, Table 21) refutes this assumption: ACT and Equal (m=0)(m = 0) produce smoother motion than RTE (jerk 1.601.60 and 1.611.61 vs. RTE's 2.952.95 and Base's 7.007.00; boundary jump 1.071.07 vs. 1.421.42 and 2.622.62), yet ACT drops to 78.9% (below unblended Base at 80.7%) and Equal stalls at 80.9% vs. RTE's 90.1%; low-pass EMA filtering similarly cuts jerk (2.592.59 and 0.640.64) while dropping success to 73.5% and 7.0% via phase lag. Instead, pass@NN (Figure 1a, Table 11) shows the frozen policy already reaches 94.6% across N=4N = 4 draws (vs. 80.7% at N=1N = 1); unlike 4×4\times same-observation averaging (Table 4, "Four samples per query": 80.3%), which stalls under shared policy error on that observation (Sec. 5.4, App. A.4), blending across successive closed-loop observations cancels both sampling noise and observation-conditioned error at 1×1\times cost (Table 20; rescuing 202 / 290 failures in Table 12).

2. Principled Drift–Variance Optimality Over Prior Heuristics (Sec. 4.3 & App. A.1–A.3, Propositions 1–3, Fig. 4a). Without an explicit drift–variance model, prior methods often made conflicting choices: ACT [1] and BAKU [2] favored oldest chunks (m<0)(m \lt 0), whereas similarity rules (CogACT's AAE [3], VOTE [4]) weighted by cosine similarity while ignoring chunk age. Modeling open-loop drift d(Δ)d(\Delta) growing monotonically with age (Fig. 1b), Props. 1–3 prove MSE-optimal weights must strictly decrease with age (m>0m \gt 0; with offset α\alpha vanishing under nested plan innovations in Prop. 3 and outperformed by pure exponential decay in App. A.1 Table 6). Flipping ACT's oldest-first direction boosts π0\pi_0 LIBERO-Long by +10.7% at s=3s = 3 (79.2% to 89.9%, Table 4) and +11.2% at s=5s = 5 (78.9% to 90.1%); moreover, simply multiplying CogACT's AAE by our recency factor (AAE + age in Table 4) lifts success from 81.1% to 88.9% (+7.8%, and +1.9% on SIMPLER in Table 2), isolating temporal recency as the missing signal in similarity aggregators.

3. Age-Adaptive Weighting by Elapsed Execution Steps (Sec. 4.2, Eq. 3, Fig. 4b). Prior implementations typically indexed weights by buffer slot kk (e.g., exp⁡(−c⋅k)\exp(-c \cdot k) in ACT [1], BAKU [2]), giving slot k=1k = 1 the same weight at s=1s = 1 and s=25s = 25. Weighting by elapsed age Δk=t−tk\Delta_k = t - t_k adapts slot k=1k = 1 under one frozen m=0.1m = 0.1 from e−0.1=0.90e^{-0.1} = 0.90 at s=1s = 1 (ESS=11.1\mathrm{ESS} = 11.1, dense variance reduction) to e−2.5=0.08e^{-2.5} = 0.08 at s=25s = 25 (ESS=1.16\mathrm{ESS} = 1.16, stale-drift suppression): even post-hoc stride tuning of unblended Freshest on π0\pi_0 LIBERO-Long peaks at 86.0% (s=10s = 10 in Fig. 4b, -4.1% below RTE at s=5s = 5, while RTE holds 88–90% up to s=20s = 20), and frozen m=0.1m = 0.1 transfers without retuning across 4 backbones, RoboTwin 2.0 (up to +8.4% / +4.6% over matched / stride-tuned Freshest, Table 3), and Franka (+7.0% / +4.3% over Freshest / Equal, Fig. 3). Moreover, unlike slot-indexed exp⁡(−c⋅k)\exp(-c \cdot k), physical age remains closed-form under variable replanning strides (DVAC, W2), asynchronous latency jitter (Δk≠k⋅s\Delta_k \neq k \cdot s, Q3), and cross-frequency scaling (m⋆=λ/fctrlm^\star = \lambda / f_{\mathrm{ctrl}}, verified zero-shot from 10/20 Hz to 50 Hz π0.5\pi_{0.5} with +4.4% four-suite gain and +9.4% on LIBERO-Long in Table 22; see Q2 in 5/6).

References:

[1] Zhao et al. "Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware." RSS, 2023.

[2] Haldar et al. "BAKU: An Efficient Transformer for Multi-Task Policy Learning." NeurIPS, 2024.

[3] Li et al. "CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation." arXiv, 2024.

[4] Lin et al. "VOTE: Vision-Language-Action Optimization with Trajectory Ensemble Voting." arXiv, 2025.

AuthorsOpenReview ↗

Response to Reviewer WiU4 (2/6)

W2: Comparison with Test-Time Scaling Baselines (RoboMonkey, TACO, BID, DVAC) and Pre-Generation Handoffs (RTC, PAINT)

In the revision (Sec. 2, Sec. 5.5 Table 5, App. C.5 Table 24 & C.7 Table 27), we cite and discuss DVAC [5] and PAINT [6] alongside RoboMonkey [7], TACO [8], BID [9], and RTC [10], and evaluate PyTorch implementations of all continuous-chunk baselines (including a 20-head TACO CFN critic trained on all 500 LIBERO-10 demos) with and without RTE (m=0.1m = 0.1) over Seeds 7–9 (500 rollouts/seed, 27,000 new sync/async rollouts) on official π0\pi_0 LIBERO-Long (H=50H = 50; eager A100-80GB latency, vs. 72.5 ms compiled in Table 10):

Table R1: Comparison and combination with test-time baselines on π0\pi_0 LIBERO-Long (3 seeds, Sec. 5.5 Table 5).

Method (Table 5)ScheduleLatency (ms)w/o RTE (%)w/ RTE (%)Gain
Standard: Bases=5,N=1s=5, N=1114.9 (1.00×)80.7±0.590.1±0.5+9.4%
MS: Consensus-Medoids=5,N=4s=5, N=4116.1 (1.01×)81.1±1.290.6±0.7+9.5%
MS: BID [9] (ICLR 2025)s=5,N=4s=5, N=4119.2 (1.04×)82.6±0.391.5±0.7+8.9%
MS: BID (call-matched) [9]s=20,N=4s=20, N=4119.2 (1.04×)86.1±0.587.8±1.4+1.7%
MS: TACO (CFN critic) [8]s=5,N=4s=5, N=4116.4 (1.01×)81.7±0.591.1±1.4+9.4%
Sync: RTC [10] (VJP)s=5,δk=0s=5, \delta_k=0202.3 (1.76×)72.7±0.485.4±0.7+12.7%
Sync: PAINT [6] (inv.)s=5,δk=0s=5, \delta_k=0203.1 (1.77×)66.1±1.580.8±0.8+14.7%
Async: Async RTC [10]s=5,δk=2s=5, \delta_k=2202.3 (1.76×)73.7±1.185.2±0.4+11.5%
Async: Async PAINT [6]s=5,δk=2s=5, \delta_k=2203.1 (1.77×)82.3±1.392.5±0.9+10.2%
Adaptive: DVAC [5]sk∈[2,10],N=1s_k\in[2,10], N=1109.3 (0.95×)83.7±0.691.3±1.2+7.6%

Note: MS = multi-sample. Synchronous δk=0\delta_k = 0 rows pin a 25-step soft (RTC) or 5-step (PAINT) prior-tail prefix, whereas native δk=2\delta_k = 2 (100 ms) rows condition only on the 2-step in-flight prefix; see 3/6 (Q4).

1. Multi-Sample Verifier Selection (RoboMonkey, TACO, BID) vs. RTE. Selection-based methods (RoboMonkey [7], TACO [8], BID [9]) typically draw M∈[4,16]M \in [4, 16] candidate chunks at observation oto_t and rank them via a verifier—a 7B VLM in RoboMonkey, a 20-head anti-exploration CFN critic in TACO, or backward coherence plus weak-policy contrast in BID—multiplying per-query FLOPs by 4×4\times–16×16\times (with task-specific verifier training in RoboMonkey and TACO). Because RoboMonkey's released 7B VLM verifier scores discrete single-step OpenVLA tokens rather than continuous 50-step flow chunks, we evaluate continuous-chunk TACO, BID, and Consensus-Medoid on π0\pi_0. Even with 4×4\times samples per query, Four-Sample averaging (80.3 ± 1.0%), Consensus-Medoid (81.1 ± 1.2%), TACO (81.7 ± 0.5%), and BID at s=5s = 5 (82.6 ± 0.3%) yield modest standalone gains—yet jump to 90.6–91.5% w/ RTE—because single-observation draws share the same step-tt perturbation (App. A.4); call-matched BID (s=20,N=4s = 20, N = 4) reaches 86.1 ± 0.5%, still 4.0% below 1x-call RTE (90.1 ± 0.5%, zero extra calls or verifier).

2. Adaptive Replanning Schedules (DVAC) vs. RTE. DVAC [5] (Denoising Tells When to Replan) monitors internal denoising variance across ODE steps to trigger replanning with adaptive stride sk∈[2,10]s_k \in [2, 10] (0.95x query cost), lifting fixed s=5s = 5 Freshest from 80.7% to 83.7 ± 0.6%, yet switches open-loop upon replanning and discards prior chunk overlap. Because RTE's weight wk∝exp⁡(−m(t−tk))w_k \propto \exp(-m(t - t_k)) depends only on query timestamps tkt_k, layering RTE (m=0.1m = 0.1) onto DVAC lifts success by +7.6% to 91.3 ± 1.2% (+1.2% over fixed-stride RTE; up to 92.8% on Seed 9), confirming that when to replan and how to blend compose synergistically.

References:

[5] Feng et al. "Denoising Tells When to Replan: Denoising-Variance Adaptive Chunking for Flow-Based Robot Policies." arXiv, 2026.

[6] Ho et al. "Start Right, Arrive Right: Asynchronous Execution via Initial Noise Selection." CoRL, 2026.

[7] Kwok et al. "RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Models." CoRL, 2025.

[8] Yang et al. "Steering Vision-Language-Action Models as Anti-Exploration: A Test-Time Scaling Approach." arXiv, 2025.

[9] Liu et al. "Bidirectional Decoding: Improving Action Chunking via Guided Test-Time Sampling." ICLR, 2025.

[10] Black et al. "Real-Time Execution of Action Chunking Flow Policies." NeurIPS, 2025.

AuthorsOpenReview ↗

Response to Reviewer WiU4 (3/6)

Q4: Pre-Generation Boundary Matching (PAINT, RTC) vs. Post-Generation RTE and Orthogonal Combination

Comparing pre-generation conditioning (PAINT [6], RTC [10]) with post-generation RTE highlights clear trade-offs in boundary geometry, variance reduction, and black-box generality, as well as strong synergy across both synchronous and native asynchronous execution:

Regime (π0\pi_0 LIBERO-Long, s=5s=5)Methodw/o RTE (%)w/ RTE (%)Gain
Sync (δk=0\delta_k=0, tail pin)RTC [10] / PAINT [6]72.7±0.4 / 66.1±1.585.4±0.7 / 80.8±0.8+12.7 / +14.7
Async delay (δk=2\delta_k=2, 100 ms)Aligned Freshest →\to RTE80.2±1.191.6±1.5+11.4%
Async delay (δk=2\delta_k=2, 100 ms)Async RTC [10] (in-flight)73.7±1.185.2±0.4+11.5%
Async delay (δk=2\delta_k=2, 100 ms)Async PAINT [6] (in-flight)82.3±1.392.5±0.9+10.2%
Async jitter (δk∈[1,3]\delta_k\in[1,3], 50–150 ms)Aligned Freshest →\to RTE81.8±0.491.0±0.4+9.2%
Async jitter (δk∈[1,3]\delta_k\in[1,3], 50–150 ms)Async RTC [10] (in-flight)73.8±1.885.3±1.7+11.5%
Async jitter (δk∈[1,3]\delta_k\in[1,3], 50–150 ms)Async PAINT [6] (in-flight)81.2±1.491.2±0.4+10.0%

1. Independent Noise Draws vs. On-Manifold Boundary Matching (and White-Box vs. Black-Box Cost). Directly answering Q4, yes: relying on independent noise draws with post-hoc convex blending means standalone RTE does not enforce on-manifold generative prefix matching. At discrete symmetric bifurcations, averaging independent draws can interpolate across modes (Appendix C.2 & F), which is precisely where pre-generation flow inversion (PAINT's 3-pass forward, reverse-Euler inversion, and Mao repainting; 1.77x latency, 203.1 ms) and VJP guidance (RTC; 1.76x, 202.3 ms) excel. Conversely, pre-generation methods require white-box access to the internal ODE velocity field, whereas RTE is purely black-box (<0.01\lt 0.01 ms overhead, 1.00x latency) and applies unchanged across flow, diffusion, and discrete/autoregressive chunk policies. Moreover, whereas PAINT and RTC are natively designed for asynchronous handoffs (δk>0\delta_k \gt 0; at δk=0\delta_k = 0, unanchored official PAINT reduces directly to Base at 80.7%), explicitly pinning a 5-step handoff prefix (PAINT) or 25-step soft prefix (RTC) to the leftover prior tail across 60 synchronous s=5s = 5 replans tends to propagate an unlucky draw's perturbation forward across chunks (66.1 ± 1.5% and 72.7 ± 0.4%), whereas RTE's independent draws achieve 1/ESS1/\mathrm{ESS} cross-chunk variance reduction (90.1 ± 0.5%).

2. Why They Combine Orthogonally + Synchronous & Native Asynchronous Synergy. Mechanistically, the two paradigms act on complementary parts of the horizon at disjoint pipeline stages: pre-generation inversion/guidance anchors the committed in-flight prefix (h<δkh \lt \delta_k) onto a consistent local trajectory branch so the arriving chunk starts on-manifold, while post-generation RTE (m=0.1m = 0.1) averages out residual trajectory noise across the unpinned future suffix (h≥δkh \ge \delta_k) as new observations arrive. Under synchronous tail pinning (δk=0\delta_k = 0), +RTE lifts RTC by +12.7% (to 85.4 ± 0.7%) and PAINT by +14.7% (to 80.8 ± 0.8%). Crucially, in their native asynchronous setting (δk=2\delta_k = 2 fixed 100 ms delay and δk∈[1,3]\delta_k \in [1, 3] 50–150 ms random jitter; Section 5.5, Appendix C.7 Table 27, and 6/6 Q3), standalone Async PAINT (pinning only the 2-step in-flight prefix while drawing fresh suffix noise) rises to 82.3 ± 1.3% and 81.2 ± 1.4%—surpassing aligned Freshest (80.2%) and synchronous Base (80.7%) at 100 ms—and layering timestamp-aligned RTE (m=0.1m = 0.1) boosts Async PAINT by +10.2% to 92.5 ± 0.9% (fixed 100 ms) and by +10.0% to 91.2 ± 0.4% (50–150 ms jitter), outperforming both standalone Async PAINT and standalone aligned RTE (91.6% / 91.0%) while likewise lifting Async RTC by +11.5% (to 85.2% / 85.3%).

References:

[6] Ho et al. "Start Right, Arrive Right: Asynchronous Execution via Initial Noise Selection." CoRL, 2026.

[10] Black et al. "Real-Time Execution of Action Chunking Flow Policies." NeurIPS, 2025.

Reviewer_7jXpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper uses episode-level pass@NN evaluations to quantify the sensitivity of generative robot policies to sampling noise. It revisits temporal ensembling through Recency-weighted Temporal Ensembling (RTE), which averages overlapping action predictions with weights that decay with age in control steps. A drift-variance analysis motivates this rule, and experiments evaluate it across four backbones, three simulation benchmarks, and five real-robot tasks.

Critical strengths
  1. The matched-stride controls establish that aggregation can improve success independently of faster replanning. On π0\pi_0 LIBERO-Long, RTE raises success from 80.7% to 90.1% at the released stride. Comparisons with alternative weighting rules and same-observation sample averaging make this the strongest strength.
  2. The evaluation extends across multiple backbones and includes 40 trials per real-world task and configuration. Transferring one decay rate across settings supports the practical value of a simple method that requires no policy retraining.
Critical weaknesses
  1. The algorithmic novelty and advantage over existing deployment choices are limited. RTE retains ACT's alignment and exponential weighting while favoring recent predictions. Uniform averaging wins in several short-horizon settings; CogACT's LIBERO averages with RTE at strides 8/4 are 93.1%/92.5%, below its released configuration's 94.6%. These results weaken the case for a broadly preferable default.
  2. The mechanistic interpretation is less secure than the performance results. Finite-N pass@NN does not cleanly separate capability from reliability, and RTE retains substantial gains with shared source noise (Table 16). The exact theoretical optimum also includes an offset that RTE drops. The analysis therefore motivates the estimator without fully explaining the observed task-level gains.
Questions for authors

None.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 7jXp (1/5)

Dear Reviewer 7jXp,

Thank you for your constructive review and for highlighting our matched-stride controls, weighting ablations, and zero-retuning transfer across 4 backbones and physical Franka hardware. We address W1 in 1/5–3/5; revisions are in blue.

W1 (1/3): Algorithmic Novelty and Advantage Over Existing Deployment Choices

As reflected in our title ("Temporal Ensembling Revisited"), we revisit a classic post-generation primitive from first principles, showing that regrounding its mechanism, theory, and age parameterization unlocks clear gains over prior deployment rules:

1. Recovering Stochastic Headroom Rather Than Trajectory Smoothing (Sec. 3, Table 21, App. A.4). Prior work (e.g., ACT [1], BAKU [2]) typically treated temporal ensembling as a heuristic for smoothing jerky motion. However, Table 21 (π0\pi_0 LIBERO-Long, s=5s = 5) shows that motion smoothness alone does not explain task success: (i) ACT's oldest-first weighting and Equal (m=0m = 0) both achieve lower jerk (1.601.60 and 1.611.61 vs. 2.952.95 for RTE and 7.007.00 for Base) and lower boundary jump ratios (1.071.07 vs. 1.421.42 for RTE and 2.622.62 for Base), yet ACT drops to 78.9% (below 80.7% Base and 90.1% RTE) and Equal stalls at 80.9%; and (ii) causal EMA on executed commands matches or lowers jerk (2.592.59 at m=0.5m = 0.5; 0.640.64 at m=0.1m = 0.1) yet drops success to 73.5% and 7.0% via phase lag. Instead, our pass@NN diagnostic (Fig. 1a, Table 11) shows the frozen policy succeeds on 94.6% of trials within N=4N = 4 draws (vs. 80.7% at N=1N = 1), and RTE recovers 90.1% at 1×1\times query rate by averaging predictions aligned to the same target step. We also show analytically (App. A.4) and empirically (Table 4) why averaging 4 fresh draws at one observation (80.3%, 4×4\times compute) stalls: same-state draws share observation-conditioned error (Table 20), whereas cross-observation blending cancels both sampling and observation error at 1×1\times compute.

2. First-Principles Drift–Variance Theory (Sec. 4.3 & App. A.2–A.3, Propositions 1–3, Fig. 4a). Without an explicit statistical model, prior methods often made conflicting heuristic choices: ACT [1] and BAKU [2] assigned the largest weight to the oldest chunk in the buffer (m<0m \lt 0 in Fig. 4a, which also drops π0\pi_0 LIBERO-Long at s=3s = 3 in Table 4 to 79.2 ± 0.2%, again below unblended Base at 80.7% and -10.7% below RTE at 89.9 ± 0.9%), whereas similarity-based rules such as CogACT's AAE [3] (81.1%) and VOTE [4] (83.9%) weighted candidates by cosine similarity while ignoring chunk age. Modeling overlapping chunks as non-identically distributed predictions whose open-loop plan drift d(Δ)d(\Delta) grows monotonically with age (Fig. 1b), we prove (Propositions 1–3) that MSE-optimal weights must decay with age (m>0m \gt 0, newest-first) to trade stale-plan bias against single-draw variance. Simply multiplying CogACT's AAE by our recency factor (AAE + age in Table 4) lifts success from 81.1% to 88.9% (+7.8%), isolating age awareness as the missing signal in similarity-based aggregators.

3. Age-Adaptive Weighting by Elapsed Execution Steps (Sec. 4.2, Eq. 3, Fig. 4b). Prior implementations (e.g., exp⁡(−c⋅k)\exp(-c \cdot k) in ACT [1], BAKU [2]) typically indexed weights by buffer slot k=0,…,K−1k = 0, \dots, K-1, giving slot k=1k = 1 the same weight whether it is 1 step old (s=1s = 1) or 25 steps old (s=25s = 25). RTE instead weights each prediction by its true elapsed age Δk=t−tk=τ+k⋅s\Delta_k = t - t_k = \tau + k \cdot s (wk∝exp⁡(−m⋅Δk)∝exp⁡(−m⋅k⋅s)w_k \propto \exp(-m \cdot \Delta_k) \propto \exp(-m \cdot k \cdot s)), so at m=0.1m = 0.1 the relative weight of k=1k = 1 adapts from e−0.1=0.90e^{-0.1} = 0.90 at s=1s = 1 (ESS=11.1\mathrm{ESS} = 11.1, maximizing variance reduction) to e−2.5=0.08e^{-2.5} = 0.08 at s=25s = 25 (ESS=1.16\mathrm{ESS} = 1.16, suppressing 25-step-old drift). Even post-hoc stride tuning of unblended Freshest on π0\pi_0 LIBERO-Long peaks at 86.0% (s=10s = 10 in Fig. 4b, -4.1% below RTE at s=5s = 5), whereas RTE with frozen m=0.1m = 0.1 holds 88–90% up to s=20s = 20 (4×4\times fewer queries) and transfers across s∈[1,50]s \in [1, 50] without retuning, gaining +3.0% on LIBERO (up to +21.5% on Long, surpassing AAE at s=8s = 8, Table 1), +1.9% over AAE (+2.8% over Equal) on SIMPLER (Table 2), up to +21.8% over Base (+8.4% over Freshest) on RoboTwin 2.0 (Table 3), and +11.8% / +7.0% / +4.3% over Base / Freshest / Equal on Franka (Fig. 3).

References:

[1] Zhao et al. "Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware." RSS, 2023.

[2] Haldar et al. "BAKU: An Efficient Transformer for Multi-Task Policy Learning." NeurIPS, 2024.

[3] Li et al. "CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation." arXiv, 2024.

[4] Lin et al. "VOTE: Vision-Language-Action Optimization with Trajectory Ensemble Voting." arXiv, 2025.

AuthorsOpenReview ↗

Response to Reviewer 7jXp (2/5)

W1 (2/3): Uniform Averaging Wins in Several Short-Horizon Settings

Uniform averaging (m=0m=0) is the zero-decay boundary of RTE (wk∝exp⁡(−mΔk)w_k \propto \exp(-m \Delta_k) at m=0m=0). We deliberately froze m=0.1m=0.1 (calibrated once at H=50H=50) globally without per-backbone or per-stride tuning; lowering mm on short horizons (H≤16H\le 16) consistently improves over m=0.1m=0.1 and moves monotonically toward 00 as H2/sH^2/s shrinks, directly validating Proposition 2:

1. Frozen m=0.1m=0.1 Across All 15 Rows of Tables 1–3 (Beats Equal in 9 / 15 Rows). Across Tables 1–3 and Franka hardware, m=0.1m=0.1 was chosen once on π0\pi_0 LIBERO-Long (H=50H=50, seed 7) and frozen globally without tuning (Sec. 5.1). Even un-tuned, frozen m=0.1m=0.1 beats Equal (m=0m=0) in our 3,200-trial Franka study (87.5% vs. 83.3%, +4.3%, p=1.3×10−4p=1.3 \times 10^{-4}) and in 9 / 15 simulation rows: 4 / 8 in LIBERO Table 1 (+5.4% / +4.7% on π0\pi_0 at s=5,3s=5, 3; +0.4% on CogACT s=8s=8; +0.3% on π0.5\pi_{0.5} s=3s=3), 1 / 1 in SIMPLER Table 2 (75.1% vs. 72.3%, +2.8%, CI [+0.4, +5.7]), and 4 / 6 in RoboTwin Table 3 (all four H=50H=50 rows on π0,π0.5\pi_0, \pi_{0.5}, up to +4.3%, p=1.7×10−5p=1.7 \times 10^{-5}; +8.0%–9.2% in Table 4). Meanwhile, Equal suffers severe stale-tail degradation below single-sample Freshest on H=50H=50 (-1.7% on LIBERO π0\pi_0 s=5s=5; -3.1% / -3.3% on RoboTwin π0/π0.5\pi_0 / \pi_{0.5} s=25s=25; -19.6% in Fig. 6a) and drops -1.9% on SIMPLER (s=1s=1), whereas in the remaining 6 short-chunk rows Equal's edge over frozen m=0.1m=0.1 is only +0.7%–1.4% (mean +1.0%).

2. Why Lowering mm on Short Chunks (H=10,16H=10, 16) Directly Confirms Proposition 2's Scaling Law. Proposition 2 (App. A.2) proves m⋆≈nΔˉ(g/σ)2∝(H2/s)(g/σ)2m^\star \approx n\bar{\Delta}(g/\sigma)^2 \propto (H^2 / s)(g/\sigma)^2, where n=H/sn=H/s active chunks have mean age Δˉ=(H−s)/2\bar{\Delta}=(H - s)/2. Under comparable 20 Hz per-step g/σg/\sigma (App. A.5), shrinking from H=50,s=5H=50, s=5 (nΔˉ=225n\bar{\Delta}=225, where m=0.1m=0.1 is optimal) to H=16,s=4→8H=16, s=4 \to 8 (nΔˉ=24→8n\bar{\Delta}=24 \to 8) and H=10,s=5H=10, s=5 (nΔˉ=5n\bar{\Delta}=5, 45×45\times smaller) shifts m⋆m^\star progressively toward 00. Lowering mm along this exact horizon trend (0.1→0.05→0.020.1 \to 0.05 \to 0.02–0.03→00.03 \to 0) improves every short-chunk row over frozen m=0.1m=0.1, beating Equal on CogACT s=8s=8 (93.2%), π0.5\pi_{0.5} s=3s=3 (97.7%), LIBERO GR00T-N1.7 s=8,4s=8, 4 (98.1%, 98.3%), and RoboTwin GR00T-N1.7 s=8s=8 (36.0%), while rising monotonically toward the m→0m \to 0 boundary on s=4s=4 (CogACT 92.5% →\to 93.1%; RoboTwin GR00T-N1.7 34.9% →\to 35.6%).

Table R1: Full 15-row comparison across Tables 1–3 (3-seed mean ± std %; bold = row-best; last column: horizon-scaled mm).

Benchmark, ModelHHssFreshestEqual (m=0)(m=0)RTE (m=0.1)(m=0.1)Scaled (m)(m)
LIBERO, CogACT16890.9±0.592.7±0.493.1±0.4 (+0.4)93.2±0.2 (0.05)
LIBERO, CogACT16481.9±0.893.4±0.692.5±0.693.1±0.4 (0.02)
LIBERO, π0\pi_050592.2±0.290.5±0.3 (-1.7)95.9±0.3 (+5.4)95.9±0.3 (0.1)
LIBERO, π0\pi_050390.3±0.691.2±0.695.9±0.4 (+4.7)95.9±0.4 (0.1)
LIBERO, π0.5\pi_{0.5}10596.8±0.197.7±0.596.7±0.197.7±0.5 (0.0)
LIBERO, π0.5\pi_{0.5}10395.8±0.797.2±0.497.5±0.5 (+0.3)97.7±0.5 (0.05)
LIBERO, GR00T-N1.716896.7±0.497.8±0.197.1±0.698.1±0.2 (0.05)
LIBERO, GR00T-N1.716496.5±0.998.0±0.096.6±0.198.3±0.1 (0.05)
SIMPLER, CogACT16174.2±1.072.3±1.1 (-1.9)75.1±0.7 (+2.8)75.1±0.7 (0.1)
RoboTwin, π0\pi_0502543.7±0.440.6±0.9 (-3.1)43.4±0.5 (+2.8)43.4±0.5 (0.1)
RoboTwin, π0\pi_0501037.7±0.844.4±0.944.7±1.0 (+0.3)44.7±1.0 (0.1)
RoboTwin, π0.5\pi_{0.5}502571.9±0.368.6±0.2 (-3.3)72.2±0.2 (+3.6)72.2±0.2 (0.1)
RoboTwin, π0.5\pi_{0.5}501074.2±0.274.5±0.278.8±0.1 (+4.3)78.8±0.1 (0.1)
RoboTwin, GR00T-N1.716830.5±0.035.9±0.235.2±0.336.0±0.3 (0.03)
RoboTwin, GR00T-N1.716426.5±0.236.2±0.734.9±0.435.6±0.4 (0.03)

Practically, Section 5.4 and Appendix C.3 (Table 19) distill this regime boundary into a zero-tuning rule: keeping m=0.1m = 0.1 when Δmax⁡>12\Delta_{\max} \gt 12 (H≥20H \ge 20) while halving to m=0.05m = 0.05 on compact short-horizon buffers (Δmax⁡≤12\Delta_{\max} \le 12) improves over m=0.1m = 0.1 in all 7 short-horizon cases and matches or exceeds Equal on 5 of 7.

AuthorsOpenReview ↗

Response to Reviewer 7jXp (3/5)

W1 (3/3): CogACT's LIBERO Averages with RTE at Strides 8/4 Are 93.1%/92.5%, Below Its 94.6% Open-Loop Release

1. Three-Seed Reproduction and RTE Gains at Released Strides s=12−16s=12{-}16 (App. B.3, Table 9). Official Dexmal/libero-db-cogact predicts H=16H=16-step chunks via 8-step DDIM and executes s=12,16,16,15s=12,16,16,15 steps open-loop on Spatial, Object, Goal, and Long. On our 3-seed harness (seeds 7/8/9, 500 eps/suite), this unmodified checkpoint reproduces 94.6±0.1%, matching the published 94.6%, and frozen RTE (m=0.1m=0.1) at those same strides matches 94.6±0.2% (Spatial 94.0±0.7%) without tuning. Because Object/Goal (s=16=Hs=16=H) have zero inter-chunk overlap and Spatial/Long (s=12,15s=12,15) overlap only on the final 1–4 tail steps (Δ∈[12,15]\Delta \in [12,15]), Table 1 evaluated s=8,4s=8,4 to test genuine full-horizon multi-chunk overlap (K=H/s=2,4K=H/s=2,4).

2. Why Unblended Replanning at s=8,4s=8,4 Drops DB-CogACT to 90.9% and 81.9% (Cross-Backbone & Per-Suite Evidence). Replanning every s<Hs \lt H steps discards the remaining 16−s16 - s actions and draws a new noisy chunk, injecting boundary jumps (jump ratio 2.622.62 for Base vs. 1.421.42 for RTE, Table 21). On backbones conditioned on proprioceptive state qtq_t and wrist RGB (π0,π0.5\pi_0,\pi_{0.5}, and GR00T-N1.7 at H=16H=16), qtq_t anchors each new chunk to the current joint/gripper state, limiting the unblended Long drop at s=4s=4–55 to -4.5% on GR00T-N1.7 (94.8% → 90.3%), -5.3% on π0\pi_0 (86.0% → 80.7%), and -2.8% on π0.5\pi_{0.5} (95.0% → 92.2%, Fig. 6a–c). By contrast, DB-CogACT takes neither qtq_t nor a wrist camera, using only one third-person RGB frame and language. Without qtq_t to anchor mid-motion replans, DDIM draws incur higher variance σ2\sigma^2 and unanchored jumps, causing a 5.7×5.7\times steeper unblended Long drop as replans increase (Fig. 6d): 91.5% (s=15s=15, 17.5 replans/ep) →\to 90.2% (s=12s=12, 21.9) →\to 82.3% (s=8s=8, 32.8) →\to 65.9% (s=4s=4, 65.7). Short suites drop only 1.8 points at s=8s=8 (93.8% vs. 95.6%), whereas 263-step Long drops 9.2 and 25.6 points at s=8,4s=8,4, pulling the 4-suite Freshest mean down to 90.9% and 81.9%.

3. Separating the Replanning Penalty from RTE's Within-Stride Recovery. The gap from 94.6% to 93.1%/92.5% thus stems primarily from switching a qtq_t-free policy to frequent unanchored replanning, not from RTE, which preserves 94.6% at the released strides. At matched strides, overlapping DDIM chunks remain noisy draws around the same motion (Eq. 4, large σ2\sigma^2, small d(Δ)d(\Delta) for Δ≤12\Delta\le 12), so RTE contracts variance and damps jumps, recovering +2.2 points over Freshest at s=8s=8 (p=0.045p=0.045, beating Equal and AAE) and +10.6 points at s=4s=4 (+21.5 on Long, p=2.2×10−4p=2.2\times 10^{-4}), while gentler mm and m=0m=0 reach 93.1%–93.4%, matching or edging out AAE (Table R2):

Table R2: DB-CogACT LIBERO success (%) (3 seeds, App. B.3, Table 9; bold = group-best Average).

ssKKMethodSpatialObjectGoalLongAverage
12–161Published93.698.095.491.494.6
12–161Open-Loop93.5±1.398.1±0.295.3±0.191.5±0.694.6±0.1
12–161–2RTE (m=0.1)(m=0.1)94.0±0.798.1±0.295.3±0.190.7±0.794.6±0.2
82Freshest93.3±0.596.4±0.491.6±1.282.3±0.590.9±0.5
82AAE92.3±0.694.9±0.292.9±0.790.4±0.492.6±0.2
82Equal (m=0)(m=0)93.3±0.494.0±0.393.3±0.790.4±0.792.7±0.4
82RTE (m=0.1)(m=0.1)93.8±0.795.8±0.592.9±1.089.7±0.593.1±0.4
82RTE (m=0.05)(m=0.05)93.8±0.495.8±0.692.4±0.890.7±0.993.2±0.2
44Freshest87.1±1.491.6±0.683.1±0.865.9±0.981.9±0.8
44AAE93.8±0.596.5±0.992.0±1.490.3±1.093.2±0.6
44Equal (m=0)(m=0)93.4±0.797.3±0.792.9±1.190.0±1.093.4±0.6
44RTE (m=0.1)(m=0.1)94.3±1.096.7±0.491.5±1.187.4±0.992.5±0.6
44RTE (m=0.02)(m=0.02)93.3±1.197.2±0.692.5±0.989.3±0.893.1±0.4

Taken together, Table R2 confirms that DB-CogACT's open-loop strength at s=12s = 12–1616 (94.4%–94.6%) stems from its qtq_t-free conditioning on synthetic LIBERO rather than a general breakdown of RTE on CogACT: once frequent replanning (s=4s = 4) introduces overlapping predictions, RTE (m=0.02)(m = 0.02) gains +11.2% over Freshest (93.1% vs. 81.9%), while in more stochastic settings frozen RTE (m=0.1)(m = 0.1) lifts CogACT to the top score on SIMPLER (75.1% vs. 74.2% Freshest and 72.3% Equal, Table 2) and physical Franka (84.5% vs. 76.0% Freshest and 77.5% Equal, p=0.0017p = 0.0017, Fig. 3 & Table 23).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

We sincerely thank all the reviewers for their constructive feedback. We appreciate the consensus on our matched-stride decomposition and ablations separating replanning from temporal blending (7jXp, WiU4, f1Fv), our first-principles drift–variance analysis (7jXp, WiU4), and RTE's training-free, client-side practicality across four VLA/diffusion backbones, three simulation benchmarks, and real-world Franka manipulation (7jXp, WiU4, f1Fv).

To thoroughly address your constructive questions and suggestions, we have conducted extensive new simulation and physical Franka evaluations alongside additional theoretical analyses, and incorporated the following major revisions into the updated manuscript (all highlighted in blue in the revised PDF):

1. Test-Time Scaling & Continuation Baselines (WiU4, f1Fv; Sec. 2 & 5.5, Table 5; App. C.5–C.6, Tables 24–26). We benchmarked and composed RTE with Consensus-Medoid, BID, TACO, DVAC, RTC, and PAINT on both standard (H=50H=50) and fine-tuned (H=30H=30, 94.8% four-suite Freshest) π0\pi_0 policies, showing that 1×-cost RTE outperforms 4× single-step selection alone and stacks orthogonally onto every baseline (+1.7 to +15.1 pts, reaching 92.5% on H=50H=50 and 94.2% on H=30H=30 LIBERO-Long).

2. Asynchronous Inference Latency & Random Jitter (WiU4, f1Fv; Sec. 5.5; App. C.7, Table 27). We evaluated timestamp-aligned RTE under 100 ms delay and 50–150 ms random jitter across LIBERO-Long and 50-task RoboTwin 2.0, holding 91.0%–91.6% on π0\pi_0 (+9.2 to +11.4 pts over Aligned Freshest) and lifting Async RTC and Async PAINT by +10.0 to +11.5 pts without retraining.

3. Horizon Regime Boundary, Control-Frequency Scaling & Theory Controls (7jXp, WiU4, f1Fv; Sec. 4.2–4.3 & 5.4, Table 4; App. A.1–A.5 & C.3, Tables 6, 19–22). We derived m⋆≈H(H−s)2s(g/σ)2m^\star \approx \frac{H(H-s)}{2s}(g/\sigma)^2 and a zero-cost rollout-log proxy η^(Δmax⁡)\hat{\eta}(\Delta_{\max}) explaining why m∈[0,0.05]m \in [0, 0.05] is optimal on short horizons (H≤16H \le 16, Table 19); verified exact inverse frequency scaling m⋆=λ/fctrlm^\star = \lambda / f_{\mathrm{ctrl}} across fine-tuned 10/20/50 Hz π0\pi_0 policies (Table 22), alongside affine/KKT/floor offset sweeps (Table 6), source-noise controls (Table 20), and ACT oldest-first and jerk controls (Tables 4, 21).

4. Paired Statistical Significance & Expanded Real-World Evaluation (7jXp, f1Fv; Abstract, Sec. 1, 5.2–5.3 & 6, Tables 1–3, Fig. 3; App. C.2, C.4 & E.3, Tables 13–15, 23, Fig. 5). We calibrated headline claims, upgraded all 50-task RoboTwin 2.0 per-task tables to 3 seeds (Tables 13–15), added paired 95% CIs and two-sided paired tt-tests across all 32 comparisons (Table 23), and expanded physical Franka evaluation from 2,400 to 3,200 paired trials with matched-stride Equal across all four backbones (87.5% RTE vs. 83.3% Equal, p=1.3×10−4p = 1.3 \times 10^{-4}).

5. Official Checkpoint & Stride-Protocol Verification (7jXp, f1Fv; Sec. 5.2 & 5.4, Fig. 4b; App. B.3 & C.2, Tables 9, 17). We verified official DB-CogACT at its released strides (94.6%, matched by frozen RTE at 94.6%, App. B.3, Table 9) versus forced mid-chunk replanning without proprioception qtq_t, reconciled π0\pi_0's code-default s=5s=5 (80.7%) versus stride-tuned s=10s=10 (86.0%, Fig. 4b), and confirmed widened RTE gains on a 60k-step RoboTwin π0\pi_0 checkpoint (Table 17).

6. Simulation-Only pass@32 Attribution (7jXp, WiU4, f1Fv; Sec. 3; App. C.1, Table 12). We clarified unbiased finite-NN estimation and extended π0\pi_0 LIBERO-Long attribution to 32 Base retries (48,000 rollouts, pass@32 = 98.2%), confirming that 98.5% (199/202) of RTE's rescues lie within unblended Base's stochastic headroom while halving regressions.

7. Symmetric-Branching Diagnostic & Action-Disagreement Gating (f1Fv; App. C.2 & F, Eq. 7, Table 16). We partitioned 50-task RoboTwin 2.0 (π0.5,s=10\pi_{0.5}, s=10) into the only 3 symmetric-branching tasks and 47 unimodal tasks (74.6% to 80.8%), and introduced an EMA-calibrated Euclidean action-disagreement gate (Eq. 7) that lifts the 3 branching tasks from 46.9% to 56.2% while preserving the 50-task mean (78.7%).

Detailed responses to each reviewer's questions are provided in the individual threads below. Thank you again for your time and thoughtful evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.