- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ReGuide, an iterative self-training framework for diffusion policies. It uses a learned dynamics model to provide phase-conditioned guidance (PCG), collects successful guided rollouts, and updates the policy through supervised learning. The authors evaluate the method on four Robomimic tasks and fine-tune a pretrained LIBERO policy separately on three tasks. The results show improvements over the base policies and, in several settings, over supervised learning baselines using successful unguided rollouts.
Critical strengths
The paper provides empirical evidence that successful guided rollouts can serve as useful training data for diffusion policies. Comparing guided and unguided data at matched numbers of retained successful trajectories helps assess the benefit of guidance beyond simply collecting additional successful demonstrations.
Critical weaknesses
-
The contribution beyond existing self-training methods is limited. Learning from generated or optimized samples is well established in STaR [1], ReST [2], PA-RL [3], and AdaptDiffuser [4]. In particular, AdaptDiffuser fine-tunes diffusion models on filtered, reward-guided trajectories, while PA-RL combines action optimization with supervised policy updates; both include robotic manipulation experiments. Although their guidance and training procedures differ from ReGuide, they limit the novelty of the overall framework.
-
The methodological contribution of PCG seems modest relative to the added complexity. Apart from its potential benefits for self-training, the guidance mechanism itself appears incremental. It combines PCA, k-means clustering, temporal ordering, phase-specific targets, distance-based gating, and phase-transition heuristics, introducing numerous hyperparameters. The paper does not sufficiently establish what this combination adds over existing guidance methods. Moreover, component ablations primarily evaluate test-time guidance rather than the policy after self-training. Downstream comparisons with simpler guidance objectives are needed to determine whether the added complexity is necessary for the reported gains.
-
The experimental scope does not establish broad applicability. The evaluation is limited to simulation and conventional diffusion policies, without experiments on larger pretrained vision-language-action models or real robots. The LIBERO fine-tuning results are particularly restricted: the authors select the three tasks with the lowest base-policy success rates and fine-tune a separate copy of the checkpoint for each task. Performance on the remaining tasks after fine-tuning is not reported, so these experiments demonstrate task-specific adaptation rather than improvement of a shared multitask policy.
[1] Zelikman, Eric, et al. "Star: Bootstrapping reasoning with reasoning." Advances in Neural Information Processing Systems 35 (2022): 15476-15488.
[2] Gulcehre, Caglar, et al. "Reinforced self-training (rest) for language modeling." arXiv preprint arXiv:2308.08998 (2023).
[3] Mark, Max Sobol, et al. "Policy agnostic rl: Offline rl and online rl fine-tuning of any class and backbone." arXiv preprint arXiv:2412.06685 (2024).
[4] Liang, Zhixuan, et al. "Adaptdiffuser: Diffusion models as adaptive self-evolving planners." arXiv preprint arXiv:2302.01877 (2023).
Questions for authors
Q1. Could the authors provide a quantitative comparison with RL methods such as EXPO [5], PA-RL[3], and DSRL [6], using the same initial policy, observation inputs, and total environment interaction budget? Including the data collected to train ReGuide’s dynamics model would help clarify its relative performance and sample efficiency.
Q2. What insight does ReGuide offer the robot learning community beyond the already-well-established "guidance and distillation" framework? AdaptDiffuser and PA-RL already demonstrate related approaches to improving generated trajectories or actions and learning from them through supervised updates. ReGuide likewise combines guidance-based trajectory improvement with a standard supervised policy update. Given these similarities, could the authors clarify what new methodological or empirical insight emerges from this combination, and what distinguishes it from prior work?
[5] Dong, Perry, et al. "Expo: Stable reinforcement learning with expressive policies." International Conference on Learning Representations. Vol. 2026. 2026.
[6] Wagenmaker, Andrew, et al. "Steering your diffusion policy with latent space reinforcement learning." arXiv preprint arXiv:2506.15799 (2025).
9 comments
(b) Hyperparameters. PCA, -means and temporal ordering are a one-time preprocessing step. , , , and are shared across tasks; , , , , the gate percentiles and vary (Appendix E). We varied , , , , and the two gate percentiles one at a time on Transport, on held-out states:
PCG sensitivity on Transport (test time, one setting changed at a time)
| Hyperparameter | Value | Success | Value | Success | Value | Success |
|---|---|---|---|---|---|---|
| # -means clusters | 20 | 0.495 ± 0.076 | 40 | 0.552 ± 0.089 | 80 | 0.507 ± 0.063 |
| PCA dimension | 64 | 0.500 ± 0.066 | 128 | 0.552 ± 0.089 | 256 | 0.508 ± 0.081 |
| Guidance scale | 0.005 | 0.523 ± 0.040 | 0.01 | 0.552 ± 0.089 | 0.02 | 0.533 ± 0.032 |
| Guided denoising steps | 20 | 0.493 ± 0.058 | 30 | 0.552 ± 0.089 | 40 | 0.477 ± 0.064 |
| Consecutive steps to advance | 1 | 0.518 ± 0.088 | 2 | 0.552 ± 0.089 | 4 | 0.533 ± 0.051 |
| Lower gate ( = 90%) | 30% | 0.470 ± 0.079 | 50% | 0.552 ± 0.089 | 70% | 0.528 ± 0.040 |
| Upper gate ( = 50%) | 70% | 0.487 ± 0.075 | 80% | 0.520 ± 0.085 | 90% | 0.552 ± 0.089 |
Default in bold; base policy 0.498 ± 0.042.
The default (bold) has the highest mean in every row, and every alternative is within one standard deviation (0.089) of it.
(c) Downstream comparison with simpler guidance. We agree with the reviewer that this is the right test. On Can and Transport, we retrained from scratch on the same number of retained successes (1N), collected without guidance, with LPB (a single global target set) or with PCG:
Downstream comparison of guidance methods (retrain from scratch, same number of retained successes)
| Task (retained successes) | Filtered BC-FS | LPB-FS | ReGuide-FS | PCG − LPB |
|---|---|---|---|---|
| Can (15, 1N) | 0.575 ± 0.040 | 0.612 ± 0.046 | 0.650 ± 0.078 | +0.038 |
| Transport (10, 1N) | 0.623 ± 0.072 | 0.648 ± 0.054 | 0.708 ± 0.088 | +0.060 |
Mean ± std over 3 seeds.
On both tasks, PCG-guided data trains the best policy, followed by LPB and then unguided data. Over unguided data, PCG adds +0.075 (Can) and +0.085 (Transport), and LPB +0.037 and +0.025. LPB also differs in the gate and the clean-action target, so this compares the two methods as a whole.
[W3] Limited experimental scope (no VLA / real robot; LIBERO task-specific)
(a) VLA models / real robots. We agree, and will state this limitation clearly in Section 6. ReGuide requires a diffusion or flow action head and a latent space in which a dynamics model can be trained; for VLAs, this could be the VLA's own intermediate features or a separate visual encoder. We have not tested either at VLA scale or on real hardware, and do not claim results there.
(b) Why the three lowest-success tasks. Self-improvement matters most where the policy fails, and the other seven tasks start at 0.73–0.95. On those seven, test-time PCG lowers success (Appendix D), which we trace to the targets and gate thresholds being pooled across the ten tasks: on tasks the policy already solves, the gate opens on rollouts that need no correction. Guided data there would largely reproduce this, so we fine-tuned where the base policy fails most. Test-time PCG is reported on all ten tasks, including the seven where it hurts.
(c) Remaining tasks after fine-tuning. Thank you for this suggestion. We evaluated each fine-tuned checkpoint on all ten tasks, on 200 held-out initial states per task (one checkpoint per target task, given the rebuttal time):
ReGuide-FT with 38 guided rollouts from the target task (bold), evaluated on all ten LIBERO tasks
| Task | Base policy | FT on K8 | FT on L5 | FT on L6 |
|---|---|---|---|---|
| Tasks we do not collect from | ||||
| Kitchen Scene 3 | 0.945 | 0.925 | 0.930 | 0.920 |
| Kitchen Scene 4 | 0.905 | 0.885 | 0.860 | 0.845 |
| Kitchen Scene 6 | 0.765 | 0.770 | 0.820 | 0.760 |
| Living Room Scene 1 | 0.870 | 0.825 | 0.870 | 0.860 |
| Living Room Scene 2a | 0.730 | 0.700 | 0.670 | 0.705 |
| Living Room Scene 2b | 0.840 | 0.825 | 0.840 | 0.855 |
| Study Scene 1 | 0.915 | 0.735 | 0.795 | 0.820 |
| Tasks ReGuide-FT is applied to | ||||
| Kitchen Scene 8 (K8) | 0.685 | 0.755 | 0.670 | 0.745 |
| Living Room Scene 5 (L5) | 0.445 | 0.455 | 0.565 | 0.475 |
| Living Room Scene 6 (L6) | 0.255 | 0.380 | 0.300 | 0.395 |
With 200 episodes per task, the evaluation noise per cell is about ±0.02–0.035.
Each checkpoint improves its target task (+0.070, +0.120, +0.140), and often the other low-success tasks as well (e.g., Living Room 6 +0.125 after fine-tuning on Kitchen 8). On the seven tasks we do not collect from, success decreases by 0.026 to 0.044 on average, most on Study Scene 1 (0.915 → 0.735–0.820). In the paper, we will report these results averaged over 3 independently trained checkpoints.
(d) Shared multitask policy. We agree that the results show task-specific adaptation of a multitask checkpoint, and we will say so in the main text.
We thank the reviewer for the detailed review and the helpful pointers to related work. We agree that the matched-data comparison is the core of the paper, and we build on it in the answers below.
[W1] Limited novelty vs. existing self-training methods
We agree that learning from self-generated, filtered samples is well established, and we do not intend to claim the generate–filter–distill loop, or success filtering, as our contribution; STaR, for example, also filters by a correctness bit. The difference is what produces the improved samples, what supervision this needs, and what is absorbed:
ReGuide's improvement step uses no reward, value function or reward model, only the demonstrations and the base policy's unlabeled rollouts; reward enters only through the final success filter. Unlike AdaptDiffuser, there is no differentiable reward, observations are images and proprioception, and the absorbed trajectories are executed rather than planned, so they need no dynamics-consistency check. Unlike PA-RL, no critic is fit. We will add this table to Section 2 to make the distinction clearer.
[W2] PCG complexity vs. benefit; ablations only at test time
(a) What PCG adds over existing guidance. On 200 held-out initial states per task, over 3 independently trained base policies:
Test-time guidance on a frozen policy
Mean ± std over 3 independently trained base policies.
On Transport PCG is above both (+0.053 over the base policy, +0.114 over LPB); on the other three tasks the three methods are within 0.025 of each other. On Transport, phase conditioning carries the gain: removing the phase order gives +0.002 over the base policy, and guiding one phase ahead gives +0.025. The ablations in Appendix D used the selection episodes; the revised paper will report them on held-out states.