- Rating
- 2
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes DS-GRPO for adapting frozen diffusion and flow-based robot policies. A lightweight steering policy samples initial noises, which are decoded into action chunks and scored by their distance to expert demonstrations. Group-relative policy gradients update the steering policy without sampler inversion, a learned critic, or backpropagation through the generator. Experiments cover offline adaptation, goal-conditioned imitation, online DAgger, and two real-world manipulation tasks.
Critical strengths
The method offers a straightforward way to learn noise-space steering directly from action-space demonstration agreement. Avoiding sampler inversion is useful when inversion is approximate or complicated by clipping in the released sampler.
Training requires only forward access to the frozen generator and updates a lightweight steering network. This makes the approach applicable when differentiating through the full sampling procedure would be inconvenient or memory-intensive.
The evaluation covers several policy backends and adaptation settings rather than a single benchmark. The comparisons with best-of-K and AWR under matched candidate counts also provide useful evidence that the observed gains are not explained solely by generating more candidates.
Critical weaknesses
The methodological contribution appears limited to combining existing noise-space steering with group-relative policy gradients and an action-distance reward. The experiments do not sufficiently establish which parts of this combination are necessary. In particular, most benchmarks use only one optimizer step per sampled batch, so PPO clipping does not affect that update under the stated procedure. A comparison with a simpler normalized REINFORCE update would help establish what the GRPO formulation contributes beyond the existing steering framework.
The efficiency comparisons do not adequately account for generator computation. For example, the OGBench comparison matches the number of updates and states while DS-GRPO decodes 32 candidates per state. DSBC can instead precompute inversion labels and reuse them during training. Without systematic comparisons of total generator calls, training time, and memory, the practical efficiency advantage remains unclear.
The online comparisons involve differences that complicate attribution of the gains. DS-GRPO receives expert action labels, whereas DSRL learns from environment rewards, so equal environment-step budgets do not imply equal supervision. FlowDAgger is a more directly comparable baseline, but its inversion procedure is modified and unsuccessful inversion labels are discarded. The paper does not report the retained-label fraction or establish how this filtering affects the comparison.
The real-world results provide limited evidence of a reliable advantage over DSBC. The reported improvement corresponds to 16 versus 13 successful pick-and-place trials out of 30, and 8 versus 7 successful insertion trials out of 10. With these small evaluation budgets and no reported uncertainty estimates, the results support feasibility more strongly than consistent superiority over the baseline.
Questions for authors
What improvement does the GRPO formulation provide over normalized REINFORCE with the same candidates and reward? Given the single optimizer step used on most benchmarks, does removing PPO clipping change the results?
How do DS-GRPO and the inversion-based baselines compare under matched total computation, including label preprocessing and all generator calls? Please report training time and peak memory alongside success rates.
What fraction of FlowDAgger inversion labels is discarded on each task? How sensitive are its results to the filtering threshold and the modified inversion procedure?
How should the comparison with DSRL be interpreted given the difference in access to expert demonstrations? Are results available that control for the expert-information budget?
Can you report uncertainty estimates and configuration-level outcomes for the real-world comparisons? Do repeated training runs or additional evaluations support a consistent advantage over DSBC?
5 comments
Response to Reviewer VpMt [2/4]
W2 / Q2: "How do DS-GRPO and the inversion-based baselines compare under matched total computation, including label preprocessing and all generator calls? Please report training time and peak memory alongside success rates."
TL;DR: We now report all of these (App. B.8). Offline, DS-GRPO makes more generator calls than DSBC, which we state in the Limitations; online, it makes fewer than FlowDAgger as we implement it. Compared with fine-tuning the backbone, it needs only forward access at inference-level memory, a property it shares with DSBC and DSRL. App. B.8 now states that this is what our efficiency claim refers to.
[New in revision] Measured with the training code of the main tables on one RTX 5090 (Tab. 16):
| Benchmark | Method | Generator calls (with backward) | FLOPs | Training time (GPU-h) | Peak memory (GiB) | Success (%) |
|---|---|---|---|---|---|---|
| OGBench cube-double (NFE per run, 100k updates × 512 states) | DSBC | 5.1×10⁸ (0), as run; 1.0×10⁷ if labels cached | 7.6×10¹⁴ | 0.23 | 0.08 | 8.8 |
| DS-BPTT | 1.6×10¹⁰ (1.6×10¹⁰) | 3.7×10¹⁶ | 0.38 | 1.09 | 19.4 | |
| DS-GRPO | 1.6×10¹⁰ (0) | 1.9×10¹⁶ | 0.30–0.33 | 0.16 | 28.2 | |
| robomimic square (decodes per 1,000 env steps, 2.5M steps) | DSRL | 4.8×10⁵ (0) | 3.4×10¹⁷ | 17.5 | 0.80 | 96.5 |
| FlowDAgger | 7,000 (5,000) | 5.2×10¹⁵ | 2.75 | 0.25 | 32.2 | |
| DS-BPTT | 2,810 (2,560) | 5.4×10¹⁴ | 2.83 | 0.26 | 53.5 | |
| DS-GRPO | 2,810 (0) | 3.2×10¹⁴ | 3.99 as run (2.79 with one fused decode call) | 0.25 | 68.2 |
- Offline, DS-GRPO costs more generator calls than DSBC. On OGBench it makes 32× the calls of DSBC as we ran it and about 1,650× those of DSBC with labels inverted once and cached (25× the FLOPs of DSBC as run); the training time is 0.30–0.33 vs. 0.23 GPU-h. As you suspected, DSBC's labels can be precomputed: our code re-inverted them per batch, which gives identical labels, and we corrected App. A.2. On LIBERO, DS-GRPO decodes 256,000 candidates per task against about 640 label inversions for DSBC.
- Matched total computation. Plotting success against cumulative generator calls on cube-double, including DSBC's label inversion, for K ∈ {2, …, 128} (Fig. 13, one seed), a larger K learns faster per iteration but not per call and does not raise the final success; at small K, DS-GRPO exceeds DSBC's final success after about 10× the calls of DSBC with cached labels, and at DSBC's as-run budget it reaches 18–27%.
- Online, DS-GRPO is cheaper than FlowDAgger as we implement it, in generator calls and FLOPs. On square it decodes 2,810 vs. 7,000 times per 1,000 environment steps, none with a backward pass (FlowDAgger: 5,000, from the refinement we use for the clamped sampler), at 16× fewer FLOPs; with the reference fixed-point inversion, which keeps about as many labels (Tab. 11), FlowDAgger would need fewer calls than DS-GRPO. As run, DS-GRPO's training time is 3.99 vs. 2.75 GPU-h for FlowDAgger; decoding all candidates of a step in one call, which gives the same decodes, brings DS-GRPO to 2.79.
- Peak memory. With π0.5, DS-GRPO trains at 7.2 GiB after torch.compile warm-up (13.1 GiB during warm-up), against 7.0 GiB at deployment; full fine-tuning with AdamW needs at least 27.9 GiB, and for StarVLA and Fast-WAM it does not fit on a 32 GB GPU (Tab. 17).
W3a / Q4: "How should the comparison with DSRL be interpreted given the difference in access to expert demonstrations?"
TL;DR: We agree that DSRL is not information-matched and now state this in the baseline description (Sec. 4.1). FlowDAgger is the supervision-matched baseline. We do not have results that control the expert-information budget.
We include DSRL as the representative noise-space steering method for online adaptation, learning from rewards rather than teacher labels. The two forms of supervision are complementary and can be combined, e.g. Flow Reversal Steering (Tang et al., 2026) adds a noise-space behavior-cloning term to the DSRL objective; we now name combining DS-GRPO with RL as future work in the Limitations.
Response to Reviewer VpMt [3/4]
W3b / Q3: "What fraction of FlowDAgger inversion labels is discarded on each task? How sensitive are its results to the filtering threshold and the modified inversion procedure?"
TL;DR: We report the kept fractions on all six tasks (App. B.5). Our inversion keeps about as many labels as the reference procedure run through the clamped sampler, and on square, loosening or removing the filter changes FlowDAgger's success by at most 5.5 points.
[New in revision] Why we modified the inversion. The DPPO sampler clamps its prediction to ±1 at every denoising step, so some expert actions have no preimage. Applied as is, the reference fixed-point solver diverges. Inverting the same teacher labels on fixed sets of 6,000 states per task (Tab. 11):
| Inversion procedure | can: kept @10⁻³ | can: kept @10⁻¹ | square: kept @10⁻³ | square: kept @10⁻¹ | Median MSE (can / square) |
|---|---|---|---|---|---|
| Reference fixed-point, ignoring the clamp | 26.5% | 32.7% | 23.4% | 24.1% | 4.5×10² / 8.8×10¹⁶ |
| Reference fixed-point, through the clamped chain | 93.1% | 98.9% | 72.7% | 93.2% | 2.0×10⁻⁴ / 3.4×10⁻⁴ |
| Ours (actor-seeded + 20 Adam steps), clamped chain | 88.9% | 100.0% | 77.0% | 97.0% | 3.1×10⁻⁴ / 3.8×10⁻⁴ |
Ignoring the clamp, the decoded actions miss the expert action by a median MSE of 4×10² on can and 9×10¹⁶ on square. Through the clamped chain, our procedure (seeded at the actor's prediction, refined with 20 Adam steps) keeps about as many labels as the reference fixed-point iteration (88.9 vs. 93.1% on can, 77.0 vs. 72.7% on square).
Kept fractions in the runs of Tab. 3 (Tab. 10):
| Benchmark | Task | Kept (%) | First 10% of training | Last 10% of training |
|---|---|---|---|---|
| Robomimic | lift | 90.5 ± 0.5 | 89.9 | 89.8 |
| Robomimic | can | 90.2 ± 0.1 | 90.4 | 89.5 |
| Robomimic | square | 74.4 ± 0.5 | 77.1 | 74.1 |
| FurnitureBench | one_leg | 69.6 ± 0.3 | 64.2 | 72.1 |
| FurnitureBench | lamp | 18.4 ± 0.1 | 6.9 | 26.2 |
| FurnitureBench | round_table | 61.3 ± 0.9 | 37.1 | 73.5 |
Sensitivity to the filter. On square, we re-ran FlowDAgger with the threshold raised to 10⁻² and 10⁻¹ and without any filter (Tab. 12):
| Threshold on decoded-action MSE | Kept (%) | Success (%) |
|---|---|---|
| 10⁻³ (Tab. 3) | 74.4 ± 0.5 | 32.2 ± 2.1 |
| 10⁻² | 88.5 ± 1.0 | 35.5 ± 0.7 |
| 10⁻¹ | 96.0 ± 0.3 | 32.8 ± 1.5 |
| None | 100 | 37.7 ± 1.6 |
| DS-GRPO (Tab. 3) | – | 68.2 ± 4.0 |
Without the filter FlowDAgger reaches 37.7%, still far below DS-GRPO's 68.2%. FlowDAgger also ends below the base policy on square, which we attribute to its deterministic noise policy behaving like the base policy at a small noise scale (Tab. 9, App. B.3). DS-GRPO needs no preimage: it scores whatever the sampler decodes, so it receives a learning signal at every visited state.
Response to Reviewer VpMt [1/4]
We sincerely thank Reviewer VpMt for the detailed and technically precise review. We appreciate your recognition that DS-GRPO learns noise-space steering directly from action-space demonstration agreement, avoids sampler inversion, needs only forward access to the generator, and is evaluated across several backbones with matched-candidate comparisons. Your questions led to four new appendix sections (B.5, B.8, B.9 and B.10); all changes are highlighted in blue.
TL;DR: You are right: with one Adam step per batch the clip is inactive, and the update is REINFORCE with a group-normalized advantage, as in the original GRPO. Removing the clip there gives bit-identical weights. The per-state group normalization appears to be what matters (with a step-size caveat below); the clip matters only when a batch is reused, which can speed up early learning but did not raise final success.
[New in revision] We now state this after Eq. (3.5), and App. B.9 ablates the update rule (3 seeds).
Success drops from 31.8 to 16.3% on cube-double and from 97.7 to 87.0% on can. The K candidates of a state share the state and the demonstration, so their mean score is a Monte Carlo estimate of that state's value, and the group advantage compares each candidate only with candidates for the same state; a batch baseline leaves differences in difficulty between states in the advantage. We do not claim this baseline as new: it is a standard variance-reduction device, and it is what makes a critic-free estimator work with demonstrations as action-space verifiers. Two caveats, now in B.9: the ablation changes centering and scaling together, and the batch baseline also moves the policy less per update, so part of the gap may come from a smaller effective step size (we did not re-tune its learning rate).
With the clip, E = 4 learns faster at first (63.5 vs. 54.5% at 0.96M steps), then stays near 60% while E = 1 keeps improving and ends higher; without the clip, E = 4 does not learn faster early but ends highest (67.5%). With three seeds, none of these differences is significant. We therefore keep one step per batch on OGBench, robomimic and FurnitureBench; LIBERO and the second-generation real-world tasks use two, which we have not ablated.