acceptodds
Under review as a conference paper at ICLR 2027

Same Objective, Different Moments: Residual-Guided Temporal Resampling for VLA Continuation

13 views

Abstract

Continued adaptation of task-adapted vision-language-action (VLA) policies seeks to improve performance under a finite update budget without inducing overfitting. Temporal resampling can reallocate this budget across trajectory moments, but without correction it also reweights the baseline continuation objective, making it difficult to determine whether performance gains arise from temporal allocation or objective reweighting. We introduce Residual-Guided Temporal Resampling (ReTime), which decouples temporal exposure from objective weighting. ReTime uses Flow-Matching residuals from a frozen starting policy to construct a non-uniform temporal proposal and applies importance correction to preserve the baseline continuation objective and its expected gradient. It modifies only the temporal sampling distribution, while keeping source and trajectory selection probabilities fixed. Across Florence and Qwen backbones in simulation and real-world settings, ReTime consistently achieves the strongest average performance. On Florence–SimplerEnv, 500 additional updates improve performance by 22.92 percentage points over the starting policy and by 11.20 points over the strongest continuation baseline. Over longer-horizon real-world continuation, ReTime continues to improve after Uniform begins to degrade. Mechanistic analysis further shows that, under matched temporal allocation, importance correction provides an additional 4.69-point gain on Florence–SimplerEnv, demonstrating that temporal exposure and objective weighting are not equivalent and motivating their explicit decoupling.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Robot Learning, Temporal Resampling, Importance Sampling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zvvmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continued adaptation, which involves continuing to train models under a limited budget for performance improvement, for VLA models. It raises the concern that increasing the sampling frequency of particular frames also increases their effective weights in the training objective, motivating the decoupling of temporal allocation from objective reweighting. It therefore introduces ReTime (Residual-Guided Temporal Resampling), which uses flow-matching residuals from a frozen starting policy to sample high-residual frames more frequently and applies importance correction to preserve the continuation objective. Experiments show that ReTime can boost performance better than uniform under both simulation and real-world experiments.

Critical strengths
  1. The idea is simple and requires minor modifications to the original training pipelines and recipes. The scoring stage is performed offline, without introducing extra inference latency or structural modifications.

  2. The authors conduct experiments in both simulation and real-world settings, providing evidence over longer training horizons. The experiments show improvements across multiple backbones and benchmarks, demonstrating the method’s effectiveness.

Critical weaknesses
  1. While this paper centers its main challenge and contribution on “decoupling temporal allocation from objective reweighing,” it doesn't sufficiently explain the significance of decoupling itself. I would like to hear more about how decoupling, as well as an unchanged continuation objective, can contribute to either general VLA performance or future VLA research.

  2. This paper conducts experiments only on GR00T-style policies implemented in StarVLA. Although different backbones are tested, I wonder whether the proposed pipeline and its performance gains can be transferred to other SOTA VLA foundation models, especially π0.5\pi_{0.5}.

Questions for authors
  1. Can you explain more about the importance of decoupling temporal allocation from objective reweighting? How could this contribute to VLA research in a broader sense?

  2. Besides GR00T, can your pipeline also be applied to other foundation models with different architectures, such as π0.5\pi_{0.5} and other flow-matching-based VLA models? Can the idea be transferred to non-flow-matching models?

  3. As the training budget is a core factor in this paper, could you explain more about why 500 additional updates were chosen for the simulation experiments and how the relative performance of the methods changes under smaller or larger training budgets? More specifically, over what range of training budgets does ReTime remain effective?

6 comments
AuthorsOpenReview ↗

Response to Reviewer zvvm (1/2)

We thank the reviewer for the thoughtful questions. We address each point below; all new results are included in the revised manuscript.


[W1 & Q1] Significance of decoupling temporal allocation from objective reweighting.

Can you explain more about the importance of decoupling temporal allocation from objective reweighting? How could this contribute to VLA research in a broader sense?

We thank the reviewer for raising this important question.

Motivation. In VLA continuation, non-uniform temporal sampling changes not only how frequently trajectory moments are encountered but also their relative contributions to the training objective. Whether such implicit objective reweighting benefits continued adaptation remains unclear, making it difficult to disentangle the effects of temporal allocation and objective modification. This motivates our central question: can we improve finite-budget VLA continuation by reallocating temporal exposure while preserving the original training objective?

Empirical evidence. Under identical temporal proposals, importance correction improves average success across four simulation settings from 31.96% to 34.93% for ReTime and from 30.74% to 32.63% for Shuffle, including a +4.69 pp gain for ReTime on Florence–SimplerEnv. Preserving the original objective thus further enhances the effectiveness of non-uniform temporal sampling.

Broader significance. This decoupling treats temporal allocation as an independent design dimension. Because the training objective remains fixed, alternative priority signals (e.g., FM residuals vs. action MSE, Table 2) and sampling strategies can be compared on equal footing, without confounding from objective changes.

Revisions. We agree that the original manuscript did not sufficiently articulate this significance and have revised:

  1. Introduction (Sec. 1): Reframed the motivation around implicit objective shifts induced by non-uniform sampling and established objective-preserving temporal allocation as the central research question.
  2. Sec. 4.3: Strengthened the interpretation of controlled experiments, highlighting the additional performance gains from importance correction under matched temporal proposals.
  3. Conclusion (Sec. 5): Clarified that these findings establish temporal allocation as a distinct design dimension in VLA continuation, enabling targeted training effort without altering the underlying learning objective.

[W2 & Q2] Generalization across VLA architectures and learning objectives.

Besides GR00T, can your pipeline also be applied to other foundation models with different architectures, such as π0.5\pi_{0.5} and other flow-matching-based VLA models? Can the idea be transferred to non-flow-matching models?

We thank the reviewer for this valuable suggestion.

Setup. We have extended ReTime to π0\pi_0-FAST, evaluating its applicability beyond GR00T-style architectures and flow-matching-based policies. Since π0.5\pi_{0.5} [1] shares the π0\pi_0 [2] architecture and its flow-matching action expert, ReTime applies to it directly. To test a non-flow-matching objective, we use π0\pi_0-FAST [3], as the public openpi release [4] supports only the flow-matching head of π0.5\pi_{0.5}. As an autoregressive VLA trained with discrete action tokens, π0\pi_0-FAST changes both the architecture and the learning objective, providing a stricter test. We evaluate Uniform, Loss-IS, and ReTime on 20 RoboTwin tasks under identical continuation budgets, with 100 rollouts per task and condition.

ConditionUniformLoss-ISReTime
Clean26.5529.4537.60
Random27.5027.4534.20
Average27.0328.4535.90

Table R1. π0\pi_0-FAST success rate (%) on 20 RoboTwin tasks (100 rollouts per task and condition).

Result. ReTime achieves 35.90% average success, exceeding Uniform and Loss-IS by +8.88 and +7.45 pp, respectively.

Why it transfers. Because the importance correction depends only on temporal sampling probabilities, not on the form of the policy's training loss, the priority signal can be adapted to the autoregressive objective (the frozen policy's token-averaged negative log-likelihood of expert action tokens) while retaining the same objective-preserving resampling mechanism. These results suggest that the benefits of objective-preserving temporal allocation are not restricted to flow-matching policies. The results and discussion have been added to Sec. 4.5 (Table 4b), with task-level results in App. C.2.

References:

  • [1] Black et al. π0.5\pi_{0.5}: A VLA Model with Open-World Generalization. CoRL 2025.
  • [2] Black et al. π0\pi_0: A VLA Flow Model for General Robot Control. arXiv:2410.24164.
  • [3] Pertsch et al. FAST: Efficient Action Tokenization for VLA Models. arXiv:2501.09747.
  • [4] openpi: github.com/Physical-Intelligence/openpi.
AuthorsOpenReview ↗

Response to Reviewer zvvm (2/2)

[Q3] Sensitivity to continuation budget.

As the training budget is a core factor in this paper, could you explain more about why 500 additional updates were chosen for the simulation experiments and how the relative performance of the methods changes under smaller or larger training budgets? More specifically, over what range of training budgets does ReTime remain effective?

Why 500 updates. Our simulation protocol deliberately targets the small-budget regime: starting from policies with basic task competence whose training losses had largely converged, 500 additional updates test how efficiently ReTime uses a limited continuation budget. We have now extended this to larger budgets.

Budget sweep. Specifically, we extended Florence–SimplerEnv continuation to 3,000 updates, evaluating ReTime and Uniform every 500 updates from the same starting checkpoint (starting-policy success: 36.46%).

Additional updates5001,0001,5002,0002,5003,000
Uniform42.1944.2746.6147.1443.2342.45
ReTime (Ours)59.3859.6460.6861.2063.0264.32
Gain (pp)+17.19+15.36+14.06+14.06+19.79+21.88

Table R2. Florence–SimplerEnv strict success (%) across continuation budgets (training seed 117465; 384 rollouts per checkpoint).

Result. ReTime consistently outperforms Uniform across the entire 500–3,000-update range, with success increasing from 59.38% to 64.32%. In contrast, Uniform peaks at 2,000 updates before declining to 42.45%. Together with the longer-horizon Florence–Phone study, these results demonstrate ReTime's sustained advantage under extended continuation, whereas additional uniform training can degrade an already well-trained policy.

Revisions. We have added this analysis to Sec. 4.4 (Fig. 7), with task-level results in Appendix C.4 (Table 16).


We hope these responses address your concerns, and would be glad to clarify any remaining points.

Thanks the authors for their response and the work put into this rebuttal, but the response does not sufficiently address my concern.

Although the π0\pi_0-FAST experiments provide evidence of transferability to non-flow-matching VLAs, they do not establish transferability to other flow-matching VLAs. Architectural compatibility alone does not establish that the performance gains transfer to π0.5\pi_{0.5}. As stated in my review, my concern is specifically about performance on SOTA VLA foundation models, given the substantial capability gap between π0.5\pi_{0.5} and π0\pi_0/π0\pi_0-FAST. The availability of only π0.5\pi_{0.5}’s flow-matching head explains the choice of π0\pi_0-FAST for the non-flow-matching evaluation, but the authors do not explain why π0.5\pi_{0.5} itself was not evaluated. This makes me doubt the generalizability of the proposed framework.

Therefore, I decide to maintain my score.

Reviewer_8e32OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReTime for continued adaptation of VLA policies. It decouples temporal exposure from objective reweighting via flow‑matching residuals and importance correction. Evaluations in simulation and real‑world settings with Florence and Qwen backbones show ReTime surpasses other baselines, validating the value of this decoupling.

Critical strengths
  1. Clean problem isolation and objective-preserving design. The paper notes that under non-uniform temporal sampling, sampling a moment more often also changes its contribution to the expected training objective. ReTime uses importance correction with weights p/q to keep the baseline continuation objective unchanged, so temporal allocation and objective reweighting stay separate. Without this correction, it is hard to tell whether gains come from better temporal allocation or from changing the objective.

  2. Solid experiments with both qualitative and quantitative analysis. ReTime achieves strong results in simulation and real-world continuation settings. The residual diagnostics and the low offline overhead further support that the method is useful.

Critical weaknesses
  1. Limited statistical and evaluation coverage relative to the claims. Most settings use only one continuation-training run, and multi-seed evidence is thin. Real-world results use only ten fixed configurations per task, so the evidence is too limited to support strong conclusions.

  2. Longer-horizon claims rest on too narrow evidence. The paper argues that ReTime’s advantage persists under longer-horizon continuation, where Uniform eventually degrades. But this evidence is limited to Florence–Phone. Broader training-horizon uncertainty remains, so this claim is not well supported yet.

Questions for authors
  1. In the continuation stage, do you train the full VLM backbone, or only the VLM interface and the action head? Table 4 lists a base learning rate, but App. B.1 mostly talks about the interface and action head.

  2. When scoring the real-world Phone and Bottle rollouts from video, did the annotators know which method each video came from?

  3. Why is downstream strict success worse at K=16 than at K=8, even though residual fidelity to Dense scoring is higher at K=16?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8e32 (1/2)

We thank the reviewer for the constructive feedback. We address each point below; all new results are included in the revised manuscript.


[W1] Limited statistical and evaluation coverage relative to the claims.

Most settings use only one continuation-training run, and multi-seed evidence is thin. Real-world results use only ten fixed configurations per task, so the evidence is too limited to support strong conclusions.

Thank you for raising this concern. We address both aspects below.

(1) Multi-seed robustness. We have extended all four simulation settings to multiple continuation-training seeds (3 for SimplerEnv and 2 for RoboTwin, due to its higher evaluation cost), and updated Table 1 to report multi-seed means, with standard deviations in App. C.1 and C.3.

MethodFlorence–SimplerEnvQwen–SimplerEnvFlorence–RoboTwinQwen–RoboTwin
Best Baseline48.61 ± 0.9937.67 ± 1.9527.95 ± 0.928.55 ± 0.21
ReTime (Ours)61.37 ± 1.9540.80 ± 0.6029.10 ± 0.1410.30 ± 0.14
Gain (pp)+12.76+3.13+1.15+1.75

Table R1. Success rate (%, mean ± std) over 3 (SimplerEnv) / 2 (RoboTwin) training seeds.

ReTime outperforms all baselines in every evaluated training-seed comparison (10/10 seed-matched comparisons: 3+3+2+2), supporting the consistency of its gains beyond a single run. We also extended key ablations to three seeds: residual alignment (App. C.3), scoring resolution (App. F.2), and proposal refresh (Table 3).

(2) Real-world evaluation coverage. ReTime outperforms the strongest baseline (Loss-IS) on all four real-world backbone–task combinations, by +16.67–17.78 pp in Phone normalized progress and +13.33–16.67 pp in Bottle full-task success:

MethodFlorence–PhoneFlorence–BottleQwen–PhoneQwen–Bottle
Loss-IS (Best Baseline)66.6756.6773.3363.33
ReTime (Ours)83.3373.3391.1176.67
Improvement (pp)+16.67+16.67+17.78+13.33

Table R2. Real-world performance (%): Phone reports normalized three-stage progress; Bottle reports full-task success.

At the configuration level, ReTime achieves 20 wins, 16 ties, and 4 losses against Loss-IS. We further conduct a leave-one-out (LOO) sensitivity analysis, excluding each configuration in turn. ReTime retains a positive margin across all four backbone–task combinations under every exclusion (minimum +7.41 pp), indicating that the gains are not driven by any single evaluated configuration. Full results are provided in Appendix C.5.3 (Table 21).


[W2] Longer-horizon claims rest on too narrow evidence.

The paper argues that ReTime's advantage persists under longer-horizon continuation, where Uniform eventually degrades. But this evidence is limited to Florence–Phone. Broader training-horizon uncertainty remains, so this claim is not well supported yet.

Thank you for this comment.

Setup. We have added a second longer-horizon setting on a different benchmark: Florence–SimplerEnv continuation extended from 500 to 3,000 updates, with ReTime and Uniform evaluated every 500 updates from the same starting checkpoint.

Method500 updatesPeak3,000 updates
Uniform42.1947.14 (at 2,000)42.45
ReTime (Ours)59.3864.32 (at 3,000)64.32

Table R3. Florence–SimplerEnv strict success (%) under extended continuation (training seed 117465; 384 rollouts per checkpoint). Full trajectory in Sec. 4.4 (Fig. 7).

Result. The Florence–Phone pattern reproduces in simulation. ReTime improves steadily and outperforms Uniform at every evaluated budget (+14.06 to +21.88 pp), whereas Uniform peaks at 2,000 updates and then falls back to 42.45%, close to its 500-update level. The longer-horizon claim is therefore now supported in both a real-world and a simulation setting.

Limitations. We acknowledge that both settings use the Florence backbone and that the simulation sweep uses a single training seed. We have revised the limitation statement in Sec. 5 accordingly, and task-level results are provided in Appendix C.4 (Table 16).

AuthorsOpenReview ↗

Response to Reviewer 8e32 (2/2)

[Q1] Trainable parameters.

In the continuation stage, do you train the full VLM backbone, or only the VLM interface and the action head? Table 4 lists a base learning rate, but App. B.1 mostly talks about the interface and action head.

Thank you for pointing out this ambiguity.

Answer. We train the full model: the entire VLM backbone (vision encoder and language model) and the action head are updated, and no module is frozen. In Table 4, "VLM interface" denoted the StarVLA module that wraps the full VLM, not a separate adapter, and "base" covers the remaining parameters outside the VLM and action head.

Revisions. We have revised App. B.1 (now App. B.3 in the revision) and the caption of Table 4 (Table 6 in the revision) to state this explicitly.


[Q2] Real-world evaluation fairness.

When scoring the real-world Phone and Bottle rollouts from video, did the annotators know which method each video came from?

Thank you for raising this concern.

Scoring protocol. The evaluation was not blinded; however, each rollout was independently scored by two evaluators against predefined, observable binary stage-completion criteria rather than subjective quality judgments, and a stage was credited only when both evaluators agreed. For example, the Phone task consists of three objectively verifiable stages: picking up the phone, placing it into the box, and closing the lid. Full-task success requires completing all three stages.

Evaluation scale. To strengthen evaluation reliability, we expanded testing from 10 to 30 rollouts per method (10 identical initial configurations, each repeated three times). We report both normalized progress and full-task success rates, with per-rollout results provided in Appendix C.5 for transparency and verification.


[Q3] Choice of KK.

Why is downstream strict success worse at K=16K = 16 than at K=8K = 8, even though residual fidelity to Dense scoring is higher at K=16K = 16?

Thank you for raising this question.

Multi-seed re-evaluation. We acknowledge that our original downstream comparison relied on a single training seed, limiting the robustness of the observed K=16K = 16 performance drop. To address this, we re-evaluated all KK settings on the same 384-rollout panel used in Table 1, across three matched training seeds. As reported in Appendix F.2, K=16K = 16 does not consistently underperform K=8K = 8: it performs better on one seed and comparably on another, with the mean difference largely driven by seed 533389. Both settings also exhibit lower cross-seed variability (standard deviations of 1.95 and 2.93) than K=4K = 4 (5.74).

MetricSeed 117465Seed 533389Seed 944964Mean
K=8K = 8 Success (%)59.3863.2861.4661.37
K=16K = 16 Success (%)62.7657.0360.9460.24
Difference (K=16K=16 − K=8K=8, pp)+3.39−6.25−0.52−1.13

Table R4. Downstream success of K=8K = 8 vs. K=16K = 16 across three training seeds.

Diminishing returns. The offline–downstream discrepancy further reflects diminishing returns in scoring resolution. Increasing KK from 4 to 8 improves recovery of Dense (Full in the paper) scoring's second-moment proxy reduction from 79.37% to 94.81%, alongside an increase in downstream success from 56.34% to 61.37%. Increasing KK further to 16 yields only 3.8 additional pp of proxy recovery while doubling scoring cost (21.1% to 42.2% of Dense), without further downstream gains.

Takeaway. Since the offline proxy is not a direct predictor of final policy success, these results suggest diminishing returns beyond K=8K = 8 rather than systematic degradation at K=16K = 16. We have clarified this relationship in the revised manuscript to further support our choice of K=8K = 8.


We hope these responses address your concerns, and would be glad to clarify any remaining points.

Reviewer_8e32OpenReview ↗

Response to the authors

Thanks for the response and additional results. I would like to keep my current score.

We sincerely thank the reviewer for their time and thoughtful consideration. We greatly appreciate their constructive feedback, which has helped us improve the manuscript.

Reviewer_72mgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continued adaptation of vision-language-action policies under a fixed extra training budget. Its main point is that changing how often different time steps are sampled usually changes both temporal exposure and the training objective, and these two effects should be separated. The proposed method, ReTime, scores time steps using residuals from a frozen starting policy, samples high-residual moments more often, and applies importance weights so that the expected continuation objective matches standard uniform continuation. The method is simple: score trajectory moments with a frozen policy, build a non-uniform temporal sampling distribution, and correct with p/q importance weights. The goal is to isolate the effect of temporal allocation itself rather than mixing it with objective reweighting. Experiments span six settings across Florence and Qwen backbones, including simulation and real-world robot tasks. ReTime achieves the best average continuation performance among the tested methods. The paper also includes helpful mechanism studies showing that both residual-guided sampling and objective preservation contribute to the gains, along with sensitivity analyses and compute discussion.

Critical strengths

The paper asks a clear and important question. The distinction between changing temporal allocation and changing the objective is meaningful, and the paper makes that point well. The method is conceptually clean and easy to follow. Using a frozen policy to score difficulty, then correcting with importance weights, is a simple and principled design. The controlled analysis is one of the strongest parts. In particular, the comparisons with shuffled sampling and with or without importance correction support the paper’s main claim better than benchmark wins alone. The empirical coverage is also solid, including multiple backbones, benchmarks, and real-world robot tasks.

Critical weaknesses

The biggest weakness is limited statistical strength. Most settings seem to rely on a single continuation run, and the real-world evaluation uses only ten fixed configurations per task. That is understandable in robotics, but it still makes the evidence less conclusive. The baseline set is reasonable but somewhat limited. I would have liked stronger comparisons to simpler residual-based resampling rules or other temporal prioritization methods at the same level. I am also not fully convinced that residuals are the best signal rather than just a useful one. The paper gives some supporting diagnostics, but the case is still somewhat indirect. Some gains are uneven across tasks, and several tasks remain difficult for all methods. So while the main idea looks promising, the practical impact appears mixed in some settings. Finally, although the objective is preserved in expectation, the actual optimization still depends on finite batches, Adam, clipping, and a frozen proposal, so the clean theory-to-practice story is only partial.

Questions for authors

How much of the gain depends on residuals specifically, compared with other frozen-policy signals? Did you try updating the proposal during continuation instead of keeping it fixed? Can you report more multi-seed results? How sensitive are the real-world results to the chosen evaluation configurations? Why does K = 16 improve offline fidelity but hurt downstream performance relative to K = 8? Since shuffled sampling can beat uniform, does some benefit come from non-uniform temporal visitation more generally? Does ReTime still help when the starting policy is weaker?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 72mg (1/3)

We thank the reviewer for the careful reading and specific questions. We address each point below; all new results are included in the revised manuscript.


[W1 & Q3] Multi-seed robustness.

Can you report more multi-seed results?

Thank you for this suggestion.

Setup. We have extended all four simulation settings to multiple continuation-training seeds (3 for SimplerEnv and 2 for RoboTwin, due to its higher evaluation cost), and updated Table 1 to report multi-seed means, with standard deviations in App. C.1 and C.3.

MethodFlorence–SimplerEnvQwen–SimplerEnvFlorence–RoboTwinQwen–RoboTwin
Best Baseline48.61 ± 0.9937.67 ± 1.9527.95 ± 0.928.55 ± 0.21
ReTime (Ours)61.37 ± 1.9540.80 ± 0.6029.10 ± 0.1410.30 ± 0.14
Gain (pp)+12.76+3.13+1.15+1.75

Table R1. Success rate (%, mean ± std) over 3 (SimplerEnv) / 2 (RoboTwin) training seeds.

Result. ReTime outperforms all baselines in every evaluated training-seed comparison (10/10 seed-matched comparisons: 3+3+2+2), supporting the consistency of its gains beyond a single run.

Additional ablations. We also extended key ablations to three seeds: residual alignment (App. C.3), scoring resolution (App. F.2), and proposal refresh (Table 3); see also Q2, Q5, and Q6 below.


[W2 & Q4] Real-world robustness.

How sensitive are the real-world results to the chosen evaluation configurations?

Thank you for this question.

Main result. ReTime outperforms the strongest baseline (Loss-IS) on all four real-world backbone–task combinations, by +16.67–17.78 pp in Phone normalized progress and +13.33–16.67 pp in Bottle full-task success:

MethodFlorence–PhoneFlorence–BottleQwen–PhoneQwen–Bottle
Loss-IS (Best Baseline)66.6756.6773.3363.33
ReTime (Ours)83.3373.3391.1176.67
Improvement (pp)+16.67+16.67+17.78+13.33

Table R2. Real-world performance (%): Phone reports normalized three-stage progress; Bottle reports full-task success.

Configuration-level analysis. At the configuration level, ReTime achieves 20 wins, 16 ties, and 4 losses against Loss-IS. We further conduct a leave-one-out (LOO) sensitivity analysis, excluding each configuration in turn. ReTime retains a positive margin across all four backbone–task combinations under every exclusion (minimum +7.41 pp), indicating that the gains are not driven by any single evaluated configuration. Full results are provided in Appendix C.5.3 (Table 21).


[W3, W4 & Q1] Residual specificity.

How much of the gain depends on residuals specifically, compared with other frozen-policy signals?

Thank you for this valuable question.

Setup. To assess whether ReTime's gains depend specifically on FM residuals, we added Action-space MSE as an alternative frozen-policy signal, scoring each timestep by the mean squared error between the frozen policy's predicted actions and expert actions.

MetricBest Baseline (FrameSkip / Loss-IS)Action MSEFM Residual (Ours)
Success (%) ↑\uparrow48.18 (185/384)57.81 (222/384)59.38 (228/384)
Gain over Baseline (pp) ↑\uparrow—+9.64+11.20

Table R3. Comparison of frozen-policy scoring signals (Florence–SimplerEnv, seed 117465, 384 rollouts).

Result. Action MSE retains approximately 86% of the improvement achieved by FM Residual over the strongest baseline. This suggests that ReTime's benefits extend beyond the specific residual formulation, highlighting the value of frozen-policy-guided temporal allocation. Consistently, on the autoregressive π0\pi_0-FAST policy, where FM residuals are unavailable, replacing them with the frozen policy's token-level NLL still yields +7.45 pp over Loss-IS (Sec. 4.5, Table 4b).

Default choice. We retain FM Residual as our default because it directly aligns with the flow-matching training objective and provides more consistent temporal rankings across Florence and Qwen (rank ρ\rho: 0.962/0.961 vs. 0.851/0.910). We have added the full Action MSE diagnostics to Table 2 and App. E.5 (Table 24) of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 72mg (2/3)

[W5] Uneven gains across tasks.

We thank the reviewer for this observation.

Scope. ReTime improves finite-budget continuation rather than guaranteeing uniform gains across tasks. Gains may be limited when baseline performance is already high (leaving little room for improvement) or when the starting policy lacks the capabilities needed for challenging tasks.

Evidence. Our new starting-policy sensitivity analysis (Sec. 4.4, Fig. 6(d)) further supports this interpretation: ReTime shows limited benefits at 5K but substantial improvements at 10K and 20K. These results help clarify the conditions under which temporal resampling is most effective. We have clarified this scope and limitation in the revised manuscript.


[W6] Practical impact and theory–practice gap.

We thank the reviewer for this observation.

Theory. ReTime guarantees objective preservation in expectation, not identical finite-step optimization trajectories. Finite batches, Adam, and clipping affect optimization dynamics, but do not negate the objective-level guarantee.

Practice. Importantly, our practical gains are empirically validated rather than inferred from theory alone. Additional multi-seed experiments confirm the robustness of our findings. In real-world evaluations, ReTime improves Phone normalized progress by 16.67–17.78 pp and Bottle full-task success by 13.33–16.67 pp over the strongest baseline, with positive gains under every leave-one-out analysis. Moreover, refreshing the proposal every 100 updates yields only +0.61 pp at an additional cost of 102.44 GPU-hours, supporting our fixed-proposal design. ReTime further extends to the autoregressive π0\pi_0-FAST policy (+7.45 pp over Loss-IS; Sec. 4.5, Table 4b) and outperforms Uniform throughout a sixfold longer continuation on Florence–SimplerEnv (500–3,000 updates; Fig. 7).

Revisions. We have clarified the scope of our theoretical guarantee and the empirical basis of our practical claims in the revised manuscript.


[Q2] Proposal refresh.

Did you try updating the proposal during continuation instead of keeping it fixed?

Thank you for this valuable suggestion.

Setup. We have added a proposal-refresh ablation to the revised manuscript (Table 3, Sec. 4.4). On Florence–SimplerEnv, we compare fixed proposals with proposals refreshed every 250 or 100 updates, using 500 continuation updates and three training seeds.

Refresh IntervalSuccess (%) ↑\uparrowJSD vs. Fixed (bits) ↓\downarrowExtra Compute (GPU-h) ↓\downarrow
Fixed (Ours)61.3700
250 updates60.686.13×10−56.13 \times 10^{-5}25.17
100 updates61.989.08×10−59.08 \times 10^{-5}102.44

Table R4. Fixed vs. refreshed proposals (Florence–SimplerEnv, 500 continuation updates, 3 seeds).

Result. The refreshed proposals remain close to the initial proposal (JSD below 10−410^{-4} bits), with no clear downstream advantage. Even refreshing every 100 updates improves success by only 0.61 pp while incurring an additional 102.44 GPU-hours.

Future work. We agree that adaptive proposal updates are a promising direction. Our study focuses on compute-efficient continuation of pretrained VLA policies with basic task competence under limited training budgets. We therefore retain the fixed proposal and have identified adaptive updates over longer continuation horizons as a direction for future work in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 72mg (3/3)

[Q5] Choice of KK.

Why does K=16K = 16 improve offline fidelity but hurt downstream performance relative to K=8K = 8?

Thank you for raising this question.

Multi-seed re-evaluation. We acknowledge that our original downstream comparison relied on a single training seed, limiting the robustness of the observed K=16K = 16 performance drop. To address this, we re-evaluated all KK settings on the same 384-rollout panel used in Table 1, across three matched training seeds. As reported in Appendix F.2, K=16K = 16 does not consistently underperform K=8K = 8: it performs better on one seed and comparably on another, with the mean difference largely driven by seed 533389. Both settings also exhibit lower cross-seed variability (standard deviations of 1.95 and 2.93) than K=4K = 4 (5.74).

MetricSeed 117465Seed 533389Seed 944964Mean
K=8K = 8 Success (%)59.3863.2861.4661.37
K=16K = 16 Success (%)62.7657.0360.9460.24
Difference (K=16K=16 − K=8K=8, pp)+3.39−6.25−0.52−1.13

Table R5. Downstream success of K=8K = 8 vs. K=16K = 16 across three training seeds.

Diminishing returns. The offline–downstream discrepancy further reflects diminishing returns in scoring resolution. Increasing KK from 4 to 8 improves recovery of Dense (Full in the paper) scoring's second-moment proxy reduction from 79.37% to 94.81%, alongside an increase in downstream success from 56.34% to 61.37%. Increasing KK further to 16 yields only 3.8 additional pp of proxy recovery while doubling scoring cost (21.1% to 42.2% of Dense), without further downstream gains.

Takeaway. Since the offline proxy is not a direct predictor of final policy success, these results suggest diminishing returns beyond K=8K = 8 rather than systematic degradation at K=16K = 16. We have clarified this relationship in the revised manuscript to further support our choice of K=8K = 8.


[Q6] Non-uniform visitation.

Since shuffled sampling can beat uniform, does some benefit come from non-uniform temporal visitation more generally?

We thank the reviewer for this observation.

Non-uniformity helps. We agree that non-uniform visitation contributes to finite-budget gains: Shuffle with importance correction (IC) outperforms Uniform across all four settings (Sec. 4.3, Fig. 5).

Shuffle vs. ReTime. Importantly, Shuffle permutes ReTime's residual-calibrated proposal masses within each trajectory (Appendix A.6), preserving their concentration while removing residual–time alignment. Although IC preserves the same expected objective, the sampling distributions induce different finite-step updates.

Importance CorrectionUniformShuffle (Non-uniform)Residual-aligned
Without IC42.1952.8654.69
With IC42.1954.6959.38

Table R6. Success rate (%) of Uniform, Shuffle, and residual-aligned sampling, with and without IC (Florence–SimplerEnv, seed 117465).

Decomposition. With IC, Shuffle gains +12.50 pp over Uniform. Beyond this, ReTime provides two complementary benefits:

  1. Residual alignment: A further +4.69 pp over Shuffle under identical proposal masses, with consistent gains across seeds, settings, and permutations (Fig. 5; Appendices C.3 and D.2).
  2. Importance correction: An additional +4.69 pp under the residual-guided proposal, while preserving the continuation objective in expectation.

[Q7] Weaker starting policies.

Does ReTime still help when the starting policy is weaker?

Thank you for raising this question.

Summary. ReTime provides limited benefits from very weak starting policies but remains effective before full task adaptation. We have added a starting-policy sensitivity analysis in Sec. 4.4 (Fig. 6(d)) and acknowledged this limitation in the revised manuscript.

Setup. We evaluate Florence–SimplerEnv checkpoints at 5K, 10K, and 20K training updates, each with the same 500-update continuation budget.

Method5K10K20K
Uniform8.0722.6642.19
Loss-IS3.1326.3048.18
ReTime (Ours)7.0335.6859.38

Table R7. Success rate (%) by starting checkpoint (Florence–SimplerEnv, seed 117465).

Result. At 5K, neither ReTime nor Loss-IS outperforms Uniform, consistent with the importance of broad temporal coverage during early-stage skill acquisition. At 10K, ReTime surpasses Uniform and Loss-IS by 13.02 and 9.38 pp, respectively, and maintains substantial gains at 20K (+17.19 and +11.20 pp).

Takeaway. These findings indicate that ReTime's effectiveness depends on starting-policy competence: its benefits emerge before full task adaptation but remain limited during early-stage skill acquisition.


We hope these responses address your concerns, and would be glad to clarify any remaining points.

Comments on the paper

2 comments

Dear Reviewers,

Please read the authors’ response and let us know whether it addresses your concerns. If any concerns remain, please briefly indicate which ones.

Thank you for your time and for your thoughtful reviews.

Best,

AC

AuthorsOpenReview ↗

Summary of Rebuttal and Revisions

We sincerely thank the AC and all reviewers for their constructive feedback. We have uploaded a revised manuscript, with changes highlighted in color. The revisions strengthen the paper in three key aspects: cross-architecture generalization, statistical and real-world robustness, and empirical validation of the proposed mechanism. In particular, we conducted new experiments on the officially released π0.5\pi_{0.5}–LIBERO checkpoint in response to the generalization concern raised in Reviewer zvvm's follow-up.

Below, we summarize how we responded to each reviewer's main concerns; details are in our individual responses.


Reviewer 72mg: Robustness and methodological justification

  • Statistical and real-world robustness (W1–W2). We expanded all four simulation settings to multiple continuation-training seeds, with ReTime outperforming the baselines in 10/10 seed-matched comparisons. In real-world evaluations, ReTime improves over the strongest baseline by +16.67–17.78 pp in Phone normalized progress and +13.33–16.67 pp in Bottle full-task success. The advantages remain positive under every leave-one-out configuration exclusion (App. C.1, C.3, C.5).
  • Residual specificity and mechanism (W3–W4, Q1, Q6). Alternative frozen-policy signals remain effective: Action MSE retains 86% of the FM-residual gain, while token-level NLL yields +7.45 pp over Loss-IS on autoregressive π0\pi_0-FAST. Controlled experiments further show +4.69 pp from residual alignment over Shuffle under identical proposal masses, supporting the value of informed temporal allocation (Table 2, Table 4b, Fig. 5, App. D.2).
  • Design choices and limitations (W5–W6, Q2, Q5, Q7). New analyses clarify proposal refresh, scoring resolution, and starting-policy competence. Refreshing proposals yields only +0.61 pp at 102.44 additional GPU-hours; three-seed results support K=8K = 8 as an effective cost–performance choice; and gains are limited for very weak starting policies. We also clarified that objective preservation is guaranteed in expectation, while practical benefits are established empirically (Table 3, Fig. 6(d), App. F.2, Sec. 5).

Reviewer 8e32: Evaluation reliability and longer-horizon evidence

  • Evaluation coverage (W1). Beyond the multi-seed results above, we expanded real-world evaluations to 30 rollouts per method, provided per-rollout outcomes, and conducted configuration-level sensitivity analyses (App. C.5).
  • Longer-horizon validation (W2). A new Florence–SimplerEnv experiment extends continuation from 500 to 3,000 updates, with ReTime outperforming Uniform at every evaluated budget. This complements the longer-horizon Florence–Phone evidence. We explicitly acknowledge that both longer-horizon settings use the Florence backbone (Fig. 7, App. C.4, Sec. 5).
  • Protocol clarification (Q1–Q3). We clarified that all model parameters are updated; that real-world scoring was not blinded, but used two evaluators applying predefined binary stage-completion criteria, with agreement required for a stage to count; and that the apparent K=16K = 16 degradation is not consistent across training seeds (App. B.3, C.5, F.2).

Reviewer zvvm: Scientific significance and generalization

  • Significance of decoupling (W1, Q1). Under identical temporal proposals, importance correction raises average success from 31.96% to 34.93% across four simulation settings. We revised the motivation and analysis to establish objective-preserving temporal allocation as a distinct design dimension (Sec. 1, 4.3, 5).
  • Evaluation on a state-of-the-art flow-matching VLA (W2, Q2, follow-up). We evaluated ReTime on the official π0.5\pi_{0.5}–LIBERO checkpoint (96.85% reported success). With identical 500-update continuation budgets, ReTime achieves 98.05% vs. 96.32% for Uniform, yielding 47% fewer failures across 6,000 evaluations. The improvement holds across three evaluation seeds (from a single continuation run) and reaches +3.73 pp on LIBERO-Long. Complementary results on autoregressive π0\pi_0-FAST (+7.45 pp over Loss-IS) provide evidence across both architectures and learning objectives (Sec. 4.5, Table 4, App. I).
  • Continuation-budget sensitivity (Q3). ReTime consistently outperforms Uniform across 500–3,000 additional updates, well beyond the original 500-update budget (Fig. 7).

In summary, the revised paper provides stronger evidence that ReTime's gains are consistent across training seeds, remain positive under real-world configuration sensitivity analyses, are supported by controlled mechanism studies, and extend to both a high-performing flow-matching VLA and an autoregressive policy. We hope these additions address the reviewers' major concerns, particularly the concern regarding generalization to state-of-the-art VLA models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.