acceptodds
Under review as a conference paper at ICLR 2027

ReGuide: From Test-Time Guidance to Self-Improving Diffusion Policies

Abstract

Behavior-cloned diffusion policies are expressive but remain vulnerable to covariate shift: small deviations from demonstrated states can compound into task failure. Existing methods address this either by expanding the training distribution through expert corrections or synthetic augmentation, or by steering a frozen policy at test time with guidance from a learned model. The former can be expensive or assumption-dependent, while the latter discards the corrected trajectories after execution. We introduce **ReGuide**, a self-improving framework that treats guided rollouts as reusable data. ReGuide first uses **Phase-Conditioned Guidance (PCG)** to generate corrective rollouts: it constructs phase-specific latent targets, applies guidance only in the drifted-but-recoverable regime, and guides through the estimated clean action. Successful guided rollouts are then absorbed back into the policy through **ReGuide-FT**, which fine-tunes the current checkpoint, or **ReGuide-FS**, which retrains from scratch on the augmented dataset; the two can also be composed and iterated. On Robomimic Can, Square, Transport, and Tool Hang, absorbing success-filtered rollouts lifts base-policy success by – from – demonstrations. At matched training-data volume, rollouts collected under PCG yield a better policy than unguided rollouts: by across three training seeds on Can, and at every rollout count on Square, so guidance improves the quality of self-collected data rather than merely its quantity. PCG's role is to generate training data rather than to improve a single rollout: at test time it matches LPB and DynaGuide on the three tasks where no guidance method separates from the base policy, and beats both on the one where any does. On LIBERO, fine-tuning a pretrained multi-task checkpoint on guided rollouts improves it on the three tasks it handles worst. Code is available at [anonymous.4open.science/r/reguide-anon-27B4](https://anonymous.4open.science/r/reguide-anon-27B4/README.md).

Primary area:
applications to robotics, autonomy, planning
Keywords:
Diffusion Policies, Test-Time Guidance, Imitation Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nzAgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ReGuide, an iterative self-training framework for diffusion policies. It uses a learned dynamics model to provide phase-conditioned guidance (PCG), collects successful guided rollouts, and updates the policy through supervised learning. The authors evaluate the method on four Robomimic tasks and fine-tune a pretrained LIBERO policy separately on three tasks. The results show improvements over the base policies and, in several settings, over supervised learning baselines using successful unguided rollouts.

Critical strengths

The paper provides empirical evidence that successful guided rollouts can serve as useful training data for diffusion policies. Comparing guided and unguided data at matched numbers of retained successful trajectories helps assess the benefit of guidance beyond simply collecting additional successful demonstrations.

Critical weaknesses
  1. The contribution beyond existing self-training methods is limited. Learning from generated or optimized samples is well established in STaR [1], ReST [2], PA-RL [3], and AdaptDiffuser [4]. In particular, AdaptDiffuser fine-tunes diffusion models on filtered, reward-guided trajectories, while PA-RL combines action optimization with supervised policy updates; both include robotic manipulation experiments. Although their guidance and training procedures differ from ReGuide, they limit the novelty of the overall framework.

  2. The methodological contribution of PCG seems modest relative to the added complexity. Apart from its potential benefits for self-training, the guidance mechanism itself appears incremental. It combines PCA, k-means clustering, temporal ordering, phase-specific targets, distance-based gating, and phase-transition heuristics, introducing numerous hyperparameters. The paper does not sufficiently establish what this combination adds over existing guidance methods. Moreover, component ablations primarily evaluate test-time guidance rather than the policy after self-training. Downstream comparisons with simpler guidance objectives are needed to determine whether the added complexity is necessary for the reported gains.

  3. The experimental scope does not establish broad applicability. The evaluation is limited to simulation and conventional diffusion policies, without experiments on larger pretrained vision-language-action models or real robots. The LIBERO fine-tuning results are particularly restricted: the authors select the three tasks with the lowest base-policy success rates and fine-tune a separate copy of the checkpoint for each task. Performance on the remaining tasks after fine-tuning is not reported, so these experiments demonstrate task-specific adaptation rather than improvement of a shared multitask policy.

[1] Zelikman, Eric, et al. "Star: Bootstrapping reasoning with reasoning." Advances in Neural Information Processing Systems 35 (2022): 15476-15488.

[2] Gulcehre, Caglar, et al. "Reinforced self-training (rest) for language modeling." arXiv preprint arXiv:2308.08998 (2023).

[3] Mark, Max Sobol, et al. "Policy agnostic rl: Offline rl and online rl fine-tuning of any class and backbone." arXiv preprint arXiv:2412.06685 (2024).

[4] Liang, Zhixuan, et al. "Adaptdiffuser: Diffusion models as adaptive self-evolving planners." arXiv preprint arXiv:2302.01877 (2023).

Questions for authors

Q1. Could the authors provide a quantitative comparison with RL methods such as EXPO [5], PA-RL[3], and DSRL [6], using the same initial policy, observation inputs, and total environment interaction budget? Including the data collected to train ReGuide’s dynamics model would help clarify its relative performance and sample efficiency.

Q2. What insight does ReGuide offer the robot learning community beyond the already-well-established "guidance and distillation" framework? AdaptDiffuser and PA-RL already demonstrate related approaches to improving generated trajectories or actions and learning from them through supervised updates. ReGuide likewise combines guidance-based trajectory improvement with a standard supervised policy update. Given these similarities, could the authors clarify what new methodological or empirical insight emerges from this combination, and what distinguishes it from prior work?

[5] Dong, Perry, et al. "Expo: Stable reinforcement learning with expressive policies." International Conference on Learning Representations. Vol. 2026. 2026.

[6] Wagenmaker, Andrew, et al. "Steering your diffusion policy with latent space reinforcement learning." arXiv preprint arXiv:2506.15799 (2025).

9 comments

We thank the reviewer for the detailed review and the helpful pointers to related work. We agree that the matched-data comparison is the core of the paper, and we build on it in the answers below.

[W1] Limited novelty vs. existing self-training methods

We agree that learning from self-generated, filtered samples is well established, and we do not intend to claim the generate–filter–distill loop, or success filtering, as our contribution; STaR, for example, also filters by a correctness bit. The difference is what produces the improved samples, what supervision this needs, and what is absorbed:

MethodWhat produces improved samplesSupervision usedWhat is absorbed
STaR [1]Sampling filtered by correctness; rationalization given the correct answerGround-truth answer (correctness filter and hint)Generated text
ReST [2]Policy sampling, filtered by scoreLearned reward modelGenerated text
PA-RL [3]Action optimization against a learned criticReward + fitted Q-functionOptimized actions
AdaptDiffuser [4]Guidance with a differentiable rewardDifferentiable reward over low-dim. stateSynthetic planned trajectories
ReGuideGuidance by a dynamics model toward phase-local demonstration latents; no rewardOne success bit per executed trajectory; no critic, no reward modelExecuted, verified recovery rollouts (images + proprio.)

ReGuide's improvement step uses no reward, value function or reward model, only the demonstrations and the base policy's unlabeled rollouts; reward enters only through the final success filter. Unlike AdaptDiffuser, there is no differentiable reward, observations are images and proprioception, and the absorbed trajectories are executed rather than planned, so they need no dynamics-consistency check. Unlike PA-RL, no critic is fit. We will add this table to Section 2 to make the distinction clearer.

[W2] PCG complexity vs. benefit; ablations only at test time

(a) What PCG adds over existing guidance. On 200 held-out initial states per task, over 3 independently trained base policies:

Test-time guidance on a frozen policy

TaskBase policyLPBPCG (ours)
Can0.570 ± 0.0260.587 ± 0.0490.595 ± 0.031
Square0.560 ± 0.0390.560 ± 0.1150.572 ± 0.077
Transport0.498 ± 0.0420.438 ± 0.0670.552 ± 0.089
Tool Hang0.027 ± 0.0190.025 ± 0.0090.035 ± 0.038

Mean ± std over 3 independently trained base policies.

On Transport PCG is above both (+0.053 over the base policy, +0.114 over LPB); on the other three tasks the three methods are within 0.025 of each other. On Transport, phase conditioning carries the gain: removing the phase order gives +0.002 over the base policy, and guiding one phase ahead gives +0.025. The ablations in Appendix D used the selection episodes; the revised paper will report them on held-out states.

(b) Hyperparameters. PCA, kk-means and temporal ordering are a one-time preprocessing step. dd, η\eta, Δ\Delta, KK and ρ\rho are shared across tasks; kk, NpN_p, MM, τ\tau, the gate percentiles and KgK_g vary (Appendix E). We varied kk, dd, η\eta, KgK_g, KK and the two gate percentiles one at a time on Transport, on held-out states:

PCG sensitivity on Transport (test time, one setting changed at a time)

HyperparameterValueSuccessValueSuccessValueSuccess
# kk-means clusters kk200.495 ± 0.076400.552 ± 0.089800.507 ± 0.063
PCA dimension dd640.500 ± 0.0661280.552 ± 0.0892560.508 ± 0.081
Guidance scale η\eta0.0050.523 ± 0.0400.010.552 ± 0.0890.020.533 ± 0.032
Guided denoising steps KgK_g200.493 ± 0.058300.552 ± 0.089400.477 ± 0.064
Consecutive steps to advance KK10.518 ± 0.08820.552 ± 0.08940.533 ± 0.051
Lower gate ℓlow\ell_{\text{low}} (ℓhigh\ell_{\text{high}} = 90%)30%0.470 ± 0.07950%0.552 ± 0.08970%0.528 ± 0.040
Upper gate ℓhigh\ell_{\text{high}} (ℓlow\ell_{\text{low}} = 50%)70%0.487 ± 0.07580%0.520 ± 0.08590%0.552 ± 0.089

Default in bold; base policy 0.498 ± 0.042.

The default (bold) has the highest mean in every row, and every alternative is within one standard deviation (0.089) of it.

(c) Downstream comparison with simpler guidance. We agree with the reviewer that this is the right test. On Can and Transport, we retrained from scratch on the same number of retained successes (1N), collected without guidance, with LPB (a single global target set) or with PCG:

Downstream comparison of guidance methods (retrain from scratch, same number of retained successes)

Task (retained successes)Filtered BC-FSLPB-FSReGuide-FSPCG − LPB
Can (15, 1N)0.575 ± 0.0400.612 ± 0.0460.650 ± 0.078+0.038
Transport (10, 1N)0.623 ± 0.0720.648 ± 0.0540.708 ± 0.088+0.060

Mean ± std over 3 seeds.

On both tasks, PCG-guided data trains the best policy, followed by LPB and then unguided data. Over unguided data, PCG adds +0.075 (Can) and +0.085 (Transport), and LPB +0.037 and +0.025. LPB also differs in the gate and the clean-action target, so this compares the two methods as a whole.

[W3] Limited experimental scope (no VLA / real robot; LIBERO task-specific)

(a) VLA models / real robots. We agree, and will state this limitation clearly in Section 6. ReGuide requires a diffusion or flow action head and a latent space in which a dynamics model can be trained; for VLAs, this could be the VLA's own intermediate features or a separate visual encoder. We have not tested either at VLA scale or on real hardware, and do not claim results there.

(b) Why the three lowest-success tasks. Self-improvement matters most where the policy fails, and the other seven tasks start at 0.73–0.95. On those seven, test-time PCG lowers success (Appendix D), which we trace to the targets and gate thresholds being pooled across the ten tasks: on tasks the policy already solves, the gate opens on rollouts that need no correction. Guided data there would largely reproduce this, so we fine-tuned where the base policy fails most. Test-time PCG is reported on all ten tasks, including the seven where it hurts.

(c) Remaining tasks after fine-tuning. Thank you for this suggestion. We evaluated each fine-tuned checkpoint on all ten tasks, on 200 held-out initial states per task (one checkpoint per target task, given the rebuttal time):

ReGuide-FT with 38 guided rollouts from the target task (bold), evaluated on all ten LIBERO tasks

TaskBase policyFT on K8FT on L5FT on L6
Tasks we do not collect from
Kitchen Scene 30.9450.9250.9300.920
Kitchen Scene 40.9050.8850.8600.845
Kitchen Scene 60.7650.7700.8200.760
Living Room Scene 10.8700.8250.8700.860
Living Room Scene 2a0.7300.7000.6700.705
Living Room Scene 2b0.8400.8250.8400.855
Study Scene 10.9150.7350.7950.820
Tasks ReGuide-FT is applied to
Kitchen Scene 8 (K8)0.6850.7550.6700.745
Living Room Scene 5 (L5)0.4450.4550.5650.475
Living Room Scene 6 (L6)0.2550.3800.3000.395

With 200 episodes per task, the evaluation noise per cell is about ±0.02–0.035.

Each checkpoint improves its target task (+0.070, +0.120, +0.140), and often the other low-success tasks as well (e.g., Living Room 6 +0.125 after fine-tuning on Kitchen 8). On the seven tasks we do not collect from, success decreases by 0.026 to 0.044 on average, most on Study Scene 1 (0.915 → 0.735–0.820). In the paper, we will report these results averaged over 3 independently trained checkpoints.

(d) Shared multitask policy. We agree that the results show task-specific adaptation of a multitask checkpoint, and we will say so in the main text.

Reviewer_44C8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ReGuide, which is a self-improving framework for diffusion policies that converts test-time guidance into reusable training data. It contains three steps:

  1. Phase-aware target construction: construct phase-specific latent targets from demonstration trajectories,
  2. Phase-conditioned guidance (PCG) and data collection: apply PCG to steer actions toward appropriate targets when the policy drifts but remains recoverable, and
  3. Iterative self-improvement learning: absorb successful guided rollouts into the policy through fine-tuning or retraining. By iteratively collecting and learning from these recovery trajectories, ReGuide improves policy performance without additional expert demonstrations. Experiments on Robomimic and LIBERO show substantial gains over the base policy and, in several settings, over filtered BC.
Critical strengths
  1. ReGuide repurposes test-time guidance as a mechanism for collecting corrective, on-policy training data. It enables diffusion policies to improve without additional expert demonstrations. This is a useful conceptual connection between test-time policy correction and IL-based policy improvement.

  2. The matched filtered-BC helps isolate whether guidance improves data quality rather than simply increasing the amount of training data. The ablations also provide evidence that phase conditioning and two-threshold gating are useful.

Critical weaknesses
  1. A fundamental weakness is that the paper does not consistently isolate the contribution of PCG from success filtering and subsequent BC updates. The advantage of guided rollouts over unguided rollouts is not stable across tasks and training modes, while most comparisons rely on a single training run. Thus, the large improvements may reflect self-training on successful rollouts rather than the proposed guidance mechanism itself.

  2. A second one is that the reported improvements are difficult to interpret because key guidance hyperparameters are selected by sweeping on the evaluation episodes. This includes the settings for PCG and, in some cases, the comparison baselines. Without a separate validation set or a fixed protocol across tasks, the reported performance may overestimate the true generalization of the method.

Questions for authors
  1. Were the PCG and baseline hyperparameters selected using a separate validation set, or were the evaluation episodes used for tuning?

  2. How much of the improvement comes from PCG itself vs simply filtering successful rollouts and applying BC? Could the authors provide matched multi-seed comparisons across some tasks and training settings?

  3. How are the number of phases, clustering settings, and phase-transition criteria selected? How sensitive are the results to these choices, and how do the authors verify that the identified phase is the correct one?

I have read the rebuttal and think the authors have addressed most of my concerns.

6 comments

We thank the reviewer for the thoughtful and clear questions. They prompted the new held-out, multi-seed experiments below, which helped us state our claims more precisely.

[W1] PCG contribution not isolated from success filtering + BC

We thank the reviewer for raising this, and we agree that a stronger test was needed. Every result here is the mean ± std over 3 independently trained checkpoints (different training seeds) rather than a single one, on 200 held-out initial states per task, for all four tasks, all rollout counts and both training modes (Q2). The two arms differ only in whether guidance was on during collection. When retraining from scratch, guided data leads in all 10 cells (+0.017 to +0.142). When fine-tuning, it leads in 8 of 10, but by a large margin only on Transport. On Square and Tool Hang most of the gain comes from filtering and BC. We will therefore state the claim more precisely: at a matched number of retained successes, guided data trains better policies when retraining from scratch, with a smaller, task-dependent effect when fine-tuning.

[W2] Hyperparameters tuned on evaluation episodes

We agree, and appreciate the reviewer pointing this out: the original numbers were selected on the evaluation episodes. On held-out states (Q1), PCG's test-time gain over the base policy is +0.053 on Transport (paper: +0.048) and +0.008 to +0.025 elsewhere. We will replace the paper's numbers with these.

[Q1] Separate validation set?

Thank you for the question. The evaluation episodes were used: PCG and DynaGuide on all tasks, and LPB on Can, were tuned on them, while LPB used its published settings elsewhere. To address this, we re-evaluated the frozen configurations on 200 held-out initial states per task, disjoint from all selection, training and data-collection states, averaging over 3 checkpoints:

Test-time guidance on a frozen policy

TaskBase policyLPBPCG (ours)
Can0.570 ± 0.0260.587 ± 0.0490.595 ± 0.031
Square0.560 ± 0.0390.560 ± 0.1150.572 ± 0.077
Transport0.498 ± 0.0420.438 ± 0.0670.552 ± 0.089
Tool Hang0.027 ± 0.0190.025 ± 0.0090.035 ± 0.038

Mean ± std over 3 independently trained base policies.

On Transport PCG is above both (+0.053 over the base policy, +0.114 over LPB); on the other three tasks the three methods are within 0.025 of each other.

[Q2] PCG vs. filtering + BC; matched multi-seed comparisons

Yes; we now provide these for all four tasks and both training modes:

Matched-data comparison, retrain from scratch

Task (base policy)Added rolloutsFiltered BC-FSReGuide-FSΔ
Can, N=15 (0.570 ± 0.026)8 (0.5N)0.567 ± 0.0240.615 ± 0.048+0.048
15 (1N)0.575 ± 0.0400.650 ± 0.078+0.075
23 (1.5N)0.603 ± 0.0250.635 ± 0.061+0.032
Square, N=30 (0.560 ± 0.039)15 (0.5N)0.607 ± 0.0710.670 ± 0.013+0.063
30 (1N)0.655 ± 0.0930.678 ± 0.055+0.023
45 (1.5N)0.660 ± 0.0560.697 ± 0.059+0.037
Transport, N=10 (0.498 ± 0.042)5 (0.5N)0.513 ± 0.0100.655 ± 0.040+0.142
10 (1N)0.623 ± 0.0720.708 ± 0.088+0.085
15 (1.5N)0.623 ± 0.0740.713 ± 0.091+0.090
Tool Hang, N=80 (0.027 ± 0.019)20 (0.25N)0.130 ± 0.0410.147 ± 0.028+0.017

Mean ± std over 3 independently trained checkpoints. Δ = ReGuide-FS − Filtered BC-FS.

Matched-data comparison, fine-tuning

Task (base policy)Added rolloutsFiltered BC-FTReGuide-FTΔ
Can, N=15 (0.570 ± 0.026)8 (0.5N)0.560 ± 0.0390.560 ± 0.035+0.000
15 (1N)0.587 ± 0.0200.595 ± 0.031+0.008
23 (1.5N)0.608 ± 0.0280.612 ± 0.044+0.004
Square, N=30 (0.560 ± 0.039)15 (0.5N)0.550 ± 0.0390.572 ± 0.058+0.022
30 (1N)0.608 ± 0.0210.592 ± 0.096−0.016
45 (1.5N)0.602 ± 0.0580.613 ± 0.040+0.011
Transport, N=10 (0.498 ± 0.042)5 (0.5N)0.520 ± 0.0960.580 ± 0.065+0.060
10 (1N)0.547 ± 0.0970.625 ± 0.041+0.078
15 (1.5N)0.558 ± 0.0750.593 ± 0.105+0.035
Tool Hang, N=80 (0.027 ± 0.019)20 (0.25N)0.112 ± 0.0100.122 ± 0.059+0.010

Mean ± std over 3 independently trained checkpoints. Δ = ReGuide-FT − Filtered BC-FT.

ReGuide-FS→FT (no matched filtered-BC baseline)

Task (base policy)Added rolloutsReGuide-FS→FT
Can, N=15 (0.570 ± 0.026)8 (0.5N)0.727 ± 0.039
15 (1N)0.667 ± 0.023
23 (1.5N)0.695 ± 0.069
Square, N=30 (0.560 ± 0.039)15 (0.5N)0.637 ± 0.038
30 (1N)0.658 ± 0.026
45 (1.5N)0.688 ± 0.064
Transport, N=10 (0.498 ± 0.042)5 (0.5N)0.660 ± 0.110
10 (1N)0.688 ± 0.010
15 (1.5N)0.722 ± 0.081
Tool Hang, N=80 (0.027 ± 0.019)20 (0.25N)0.182 ± 0.101
40 (0.5N)0.163 ± 0.040
60 (0.75N)0.175 ± 0.005
80 (1N)0.135 ± 0.056

When retraining from scratch, guided data leads in all 10 cells. When fine-tuning, it leads at all three Transport counts (+0.060, +0.078, +0.035), and on the other tasks the two arms are within 0.022 of each other. ReGuide-FS→FT reaches 0.667–0.727 on Can but has no matched baseline, so it does not isolate guidance. Relative to the base policy, filtering and BC account for most of the gain on Square and Tool Hang, while guidance adds the most on Can and Transport; we agree with the reviewer that filtering and BC are a substantial part of the gain, which is why these comparisons hold them fixed.

[Q3] Phase number, clustering, transition criteria; sensitivity; verifying phase correctness

(a) Selection. Clusters are ordered by the mean timestep of their members and grouped into NpN_p contiguous phases. The phase advances when the predicted latent is closer to the targets of phase j+1j+1 than to those of phase jj by a margin Δ\Delta, for KK consecutive steps. The task-specific values (kk, NpN_p, MM, τ\tau, the gate percentiles and KgK_g) were swept per task on the evaluation episodes (Appendix E). d=128d=128, η=0.01\eta=0.01, Δ=0.2\Delta=0.2 and K=2K=2 are shared across all tasks: Δ\Delta was fixed in advance, and for dd, η\eta and KK we only briefly checked one or two alternatives; kk is 30 or 40, and NpN_p is 3 (Can, Square) or 4 (Transport, Tool Hang).

(b) Sensitivity. On Transport, one hyperparameter at a time, on held-out states:

PCG sensitivity on Transport (test time, one setting changed at a time)

HyperparameterValueSuccessValueSuccessValueSuccess
# kk-means clusters kk200.495 ± 0.076400.552 ± 0.089800.507 ± 0.063
PCA dimension dd640.500 ± 0.0661280.552 ± 0.0892560.508 ± 0.081
Guidance scale η\eta0.0050.523 ± 0.0400.010.552 ± 0.0890.020.533 ± 0.032
Guided denoising steps KgK_g200.493 ± 0.058300.552 ± 0.089400.477 ± 0.064
Consecutive steps to advance KK10.518 ± 0.08820.552 ± 0.08940.533 ± 0.051
Lower gate ℓlow\ell_{\text{low}} (ℓhigh\ell_{\text{high}} = 90%)30%0.470 ± 0.07950%0.552 ± 0.08970%0.528 ± 0.040
Upper gate ℓhigh\ell_{\text{high}} (ℓlow\ell_{\text{low}} = 50%)70%0.487 ± 0.07580%0.520 ± 0.08590%0.552 ± 0.089

Default in bold; base policy 0.498 ± 0.042.

The default (bold) has the highest mean in every row, and every alternative is within one standard deviation (0.089) of it. The margin Δ\Delta is applied in every run; varying it over 0.1, 0.2 and 0.5 gave the same success on Transport (0.552 at each value). The number of phases NpN_p and the number of targets per phase MM are swept in Tables 5 and 6 of the paper.

Reviewer_nnMLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReGuide turns test-time guidance into training data for diffusion policies. Its Phase-Conditioned Guidance (PCG) constructs phase-specific latent targets, restricts guidance to an intermediate distance regime, and updates the estimated clean action. Successful guided rollouts are then incorporated through fine-tuning, training from scratch, or their composition. The central contribution is evidence that guided rollouts can improve subsequent behavior cloning beyond collecting the same number of successful unguided trajectories. Experiments cover four Robomimic tasks and task-specific adaptation of a pretrained LIBERO policy.

Critical strengths
  1. The matched-data comparisons directly test whether guidance improves the usefulness of collected trajectories. On Can, the three-seed ReGuide-FS result of 0.6737 versus 0.5845 for Filtered BC-FS provides meaningful evidence beyond simply collecting more successful demonstrations.

  2. The phase-conditioned targets, restricted guidance regime, and subsequent supervised update form a well-motivated pipeline. Reporting the weaker Transport evidence and negative LIBERO guidance results helps define a useful, task-dependent contribution rather than suggesting universal improvement.

Critical weaknesses
  1. Selection and reporting use the same evaluation episodes. Section 5 and Appendix E state that PCG hyperparameters are selected by maximizing success over the same 2,500 episodes used for reporting. Table 2 similarly selects each method's best rollout count. Disclosure is welcome, but the resulting estimates remain optimistic. This bias need not cancel between methods: PCG has additional selection dimensions, while LPB uses published settings on several tasks. Even equal numbers of candidates do not guarantee equal selection bias. This particularly affects small performance differences. A focused evaluation of the already selected configurations on unused initial states would substantially strengthen the key comparisons without requiring a new benchmark.

  2. The background guidance equations reverse the stated cost-minimization direction. Equation (2) subtracts a positive multiple of the distance gradient from the noise prediction. Substituting this into Equation (1) adds a positive multiple of that gradient to the next action iterate, although Section 3 defines the objective as distance to the target. Equation (7) in Appendix B repeats this issue. In contrast, Equation (4) explicitly descends the clean-action objective, followed by conversion back to a noise prediction in Equation (5). Thus, the inconsistency establishes a mathematical presentation error, but does not establish that PCG itself was implemented incorrectly. The implemented updates for both PCG and the rerun baselines should be specified.

Questions for authors
  1. With hyperparameters and rollout counts fixed, are independent evaluation results available for the Can matched-data FS comparison and the Transport test-time guidance comparison?

  2. Do the implementations follow Equations (4)–(5), and are the signs in Equations (2) and (7) exclusively manuscript errors? Are any baseline results affected?

3 comments

We sincerely thank the reviewer for the careful reading, for catching the sign error, and for suggesting the held-out evaluation, which we have now run.

[W1] Selection and reporting use the same evaluation episodes

We agree with the reviewer, and thank them for this concrete suggestion. We froze the selected configurations and re-evaluated them on 200 held-out initial states per task, disjoint from the 2,500 selection episodes and from all training and data-collection states. Within a task all methods share these states, every result is averaged over 3 independently trained checkpoints, every rollout count is reported, and nothing was re-selected. The results are in Q1: guided data still leads unguided data in every retrain-from-scratch comparison, and at test time PCG is above LPB and the base policy on Transport, while on the other three tasks the three methods are within 0.025 of each other. We will gladly replace the paper's numbers with these.

Table 2 (LIBERO) likewise reports each arm at its best rollout count. All four counts are reported in Table 17; averaged over counts, ReGuide-FT is above Filtered BC-FT on all three tasks (+0.023, +0.042, +0.027 on Living Room 5, Living Room 6 and Kitchen 8), though not at every count (8 of 12 task–count cells). We also re-evaluated ReGuide-FT at 38 rollouts on 200 held-out initial states (one checkpoint per task, given the rebuttal time): Kitchen 8 0.685 → 0.755, Living Room 5 0.445 → 0.565, Living Room 6 0.255 → 0.395.

[W2] Sign error in guidance equations (2) and (7)

Thank you for spotting this; it is indeed a manuscript error. Since δ\delta is a cost, Equations (2) and (7) should read

ϵ^(Atk)=ϵθ(Atk)+η1−αˉk∇Atkδ(Atk)\hat\epsilon(A_t^k) = \epsilon_\theta(A_t^k) + \eta \sqrt{1-\bar\alpha_k} \nabla_{A_t^k} \delta(A_t^k)

which gives a descent step through Equation (1). Equation (6) is correct as written, since it uses a reward rr. The implementation itself follows the intended direction: PCG implements Equations (3)–(5), and LPB and DynaGuide take the gradient with respect to the noisy iterate AtkA_t^k (Section 4.2) in the descent direction. Following the reviewer's suggestion, we will state each method's update explicitly in Appendix B.

[Q1] Independent evaluation with fixed hyperparameters

Yes, and we thank the reviewer for asking. Under the protocol described in W1:

Matched-data comparison, retrain from scratch

Task (base policy)Added rolloutsFiltered BC-FSReGuide-FSΔ
Can, N=15 (0.570 ± 0.026)8 (0.5N)0.567 ± 0.0240.615 ± 0.048+0.048
15 (1N)0.575 ± 0.0400.650 ± 0.078+0.075
23 (1.5N)0.603 ± 0.0250.635 ± 0.061+0.032
Square, N=30 (0.560 ± 0.039)15 (0.5N)0.607 ± 0.0710.670 ± 0.013+0.063
30 (1N)0.655 ± 0.0930.678 ± 0.055+0.023
45 (1.5N)0.660 ± 0.0560.697 ± 0.059+0.037
Transport, N=10 (0.498 ± 0.042)5 (0.5N)0.513 ± 0.0100.655 ± 0.040+0.142
10 (1N)0.623 ± 0.0720.708 ± 0.088+0.085
15 (1.5N)0.623 ± 0.0740.713 ± 0.091+0.090
Tool Hang, N=80 (0.027 ± 0.019)20 (0.25N)0.130 ± 0.0410.147 ± 0.028+0.017

Mean ± std over 3 independently trained checkpoints. Δ = ReGuide-FS − Filtered BC-FS.

Can matched-data FS. ReGuide-FS leads Filtered BC-FS at every count (+0.048, +0.075, +0.032), and guided data leads in all 10 cells. The 1N margin is +0.075, against +0.089 in the paper.

Test-time guidance on a frozen policy

TaskBase policyLPBPCG (ours)
Can0.570 ± 0.0260.587 ± 0.0490.595 ± 0.031
Square0.560 ± 0.0390.560 ± 0.1150.572 ± 0.077
Transport0.498 ± 0.0420.438 ± 0.0670.552 ± 0.089
Tool Hang0.027 ± 0.0190.025 ± 0.0090.035 ± 0.038

Mean ± std over 3 independently trained base policies.

Transport test-time guidance. PCG reaches 0.552, against 0.438 for LPB and 0.498 for the base policy: +0.053 over the base policy (paper: +0.048), and PCG is above the base policy on each of the 3 checkpoints (+0.010, +0.105, +0.045). On the other three tasks the three methods are within 0.025 of each other. Due to the limited time of the rebuttal period, we were not yet able to re-run DynaGuide; the revised paper will report all methods under this protocol.

[Q2] Implementation vs. Equations (4)–(5); baselines affected?

Yes. PCG follows Equations (4)–(5), and the signs in (2) and (7) are manuscript errors only. No baseline results are affected, since the implemented baselines descend the cost (W2).

I have no further questions.

We thank the reviewer for the careful review and the constructive suggestions, in particular the held-out evaluation and catching the sign error. We will incorporate the new results and corrections into the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.