- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work presents QPILOT, which harnesses the whole denoising process instead of just on one-step setting in diffusion-based policy's RL. The proposed method is claimed to help diffusion-based policies' RL out-perform their fine-tuning.
Critical strengths
- The idea of steering denoising in all steps, is clearly shown and illustrated, and easy to follow.
- Comprehensive experiments are provided to justify the proposed method.
Critical weaknesses
- Experiment Fairness. QPILOT is tested on >1 NFE, while some of the comparisons are done on 1 NFE. It would be good to add comparisons with the same NFE when possible, and ablate QPILOT on NFEs.
- 1-NFE benefits. Most robotics tasks are latency sensitive. However, if I understand correctly, the QPILOT would have 0 impact on 1-NFE model. It would be interesting to see whether RL with QPILOT and then distill to 1 NFE would help.
Questions for authors
See critical weaknesses. Besides: (1) It will be great if all 50 task results in Table 1 can be released.
9 comments
W2: "Most robotics tasks are latency sensitive ... QPILOT would have 0 impact on 1-NFE model ... see whether RL with QPILOT and then distill to 1 NFE would help."
We agree that steering a standard flow policy has little effect at one step. Run for a single step, a flow-matching policy outputs roughly the average data action for the state. On OGBench's play datasets, which mix many behaviors, that average is often not a valid action. That is why the 10-step flow baselines also collapse at one step (QAM-E: 3 28 at ).
Try distilling to 1 NFE. We thank the reviewer for this suggestion, which works remarkably well as a complement to QPILOTS rather than a replacement: the student needs QPILOTS-U as its teacher during training, and in exchange for one-step inference it gives up about 5 online points (85 vs. 90). For a fixed base policy and critic, the QPILOTS-U sampler maps each noise input to one action, so a one-step student can regress the steered action for the same noise. This is FQL's one-step distillation with the steered sampler as teacher. We trained the student online alongside QPILOTS-U; it is never used to act or to compute critic targets. On 16 tasks across 8 domains (4 seeds), the student keeps about 95% of its teacher's success at one network pass per action. It outperforms every one-step method, including QAM-F, QAM's own variant that distills its flow into a one-step actor, and even beats 10-step QAM-E (App. F.3, Table 11):
| 16 tasks, success off on | 1 network pass | 10 flow steps |
|---|---|---|
| ReBRAC | 45 65 | |
| FQL | 37 79 | |
| QAM-F (QAM distilled to one step) | 46 79 | |
| QAM-E | 43 82 | |
| QPILOTS-U: distilled student / its teacher | 50 85 | 53 90 |
Steering during training and distilling for deployment therefore gives a fast policy at deployment time. At batch size 1 on one NVIDIA L4 GPU, the student takes 0.27 ms per action, against 1.56 ms for its QPILOTS-U teacher (5.9× faster) and 0.53 ms for the unsteered 10-step base flow, on humanoidmaze-large; on cube-triple the student is 5.6× faster than its teacher (0.28 vs. 1.59 ms). We are running the student on all 50 tasks with 8 seeds and aim to add it as a full variant to Table 1 by the camera-ready.
Q1: "It will be great if all 50 task results in Table 1 can be released."
App. Table 19 now lists all 50 tasks for all 13 methods of Table 1, and we post it as a follow-up comment. Baselines are averaged over the 12 seeds released by the QAM authors, ours over 8 seeds. Per-task learning curves are in App. Fig. 11.
I thank the authors for the rebuttal and further evidences! For W1, I thank the author for listing the NFEs, and I suggest including them in the main results. I notice that the 10 NFE methods also have some diffs. It would be nice to list the latency of each of them, given the same compute environment, to further justify the comparison.
We thank the reviewer for the review, for recognizing the comprehensiveness of our experiments, and for the suggestion to distill QPILOTS into a one-step policy, which worked remarkably well as a complement to steering: the student learns from QPILOTS' steered actions and keeps most of its gains. We ran additional experiments that directly address each concern: (1) on NFE fairness, nine of the eleven Table 1 baselines, including the strongest (QAM-E), use the same 10-step sampler as QPILOTS, and our new NFE ablation (§5.2, Fig. 3) shows that at a matched 3 steps QPILOTS-U outperforms QAM-E by 43 online points (75% vs. 32% on ten tasks); (2) for one-step deployment, we distilled QPILOTS-U into a one-step student that runs 5.6–5.9× faster per action, keeps about 95% of its teacher's success, and outperforms every one-step method, including QAM's own distilled variant, and even 10-step QAM-E (85% vs. 82% online on 16 tasks); and (3) we release per-task results for all 50 tasks and all 13 methods (App. Table 19). All changes are shown in blue in the revised submission.
W1: Fairness: "QPILOT is tested on >1 NFE, while some of the comparisons are done on 1 NFE ... add comparisons with the same NFE ... and ablate QPILOT on NFEs."
(NFE: network evaluations per action.) Only two Table 1 baselines act with a single network pass: ReBRAC (a Gaussian actor) and FQL (a distilled one-step actor). The other nine use the same 10-step Euler sampler as QPILOTS. The baseline closest in compute is CGQL, which also takes a critic gradient at every step, and QPILOTS-U is 22 points higher online:
NFE-matched comparison and ablation. The QAM authors released QAM-E runs with 1, 3, 20 and 30 flow steps on one task per domain. We trained QPILOTS-U with K=3 steps on all ten of these tasks (4 seeds), keeping the α tuned at 10 steps. Mean success on these ten tasks, off → on:
Going from 10 to 3 steps, QAM-E loses 54 online points and fails on most of these tasks, while QPILOTS-U loses 19. Each steered step adds one backward pass, so QPILOTS-U at K=3 costs 3 flow evaluations plus 2 gradient passes. §5.2 now has this as a fourth design axis: Fig. 3 plots success against K on the eight of these tasks that also have the distilled student (below), and App. F.3 (Fig. 8) gives the per-task curves and the K=3 mean over all 20 tuning tasks.