acceptodds
Under review as a conference paper at ICLR 2027

DS-GRPO: Noise-Space Adaptation of Diffusion Policies with Action-Space Verification

11 views

Abstract

Diffusion steering adapts a pretrained generative policy without altering its weights: a small steering policy chooses the initial noise of the frozen sampler, re-weighting the base policy's behaviors within their support. Yet the signals one could adapt to, e.g., a demonstration, a success, a distance to a target, are all only verifiable in action space, while steering optimizes in noise space. Existing methods resolve this mismatch with different trade-offs. Critic-based noise-space RL verifies in action space but pays the cost of exploration and value estimation through environment transitions. Inversion-based steering instead maps demonstrated actions back to noise space, regressing on noise labels rather than minimizing action-space distance. We propose **DS-GRPO**, combining the benefits of both supervision from demonstrations and direct verification in action space. DS-GRPO samples a group of noises from the steering policy, decodes them through the frozen sampler, and uses the demonstration as an action-space verifier to score the resulting actions. These scores guide steering-policy updates through group-relative policy gradients. Across simulation and real-world experiments including offline behavior cloning, goal-conditioned behavior cloning, and online DAgger, DS-GRPO matches or outperforms pre-existing steering methods on most tasks, providing a concrete step towards efficient yet effective policy adaptation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Adaptation, Manipulation, Diffusion Steering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VpMtOpenReview ↗
Rating
2
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DS-GRPO for adapting frozen diffusion and flow-based robot policies. A lightweight steering policy samples initial noises, which are decoded into action chunks and scored by their distance to expert demonstrations. Group-relative policy gradients update the steering policy without sampler inversion, a learned critic, or backpropagation through the generator. Experiments cover offline adaptation, goal-conditioned imitation, online DAgger, and two real-world manipulation tasks.

Critical strengths

The method offers a straightforward way to learn noise-space steering directly from action-space demonstration agreement. Avoiding sampler inversion is useful when inversion is approximate or complicated by clipping in the released sampler.

Training requires only forward access to the frozen generator and updates a lightweight steering network. This makes the approach applicable when differentiating through the full sampling procedure would be inconvenient or memory-intensive.

The evaluation covers several policy backends and adaptation settings rather than a single benchmark. The comparisons with best-of-K and AWR under matched candidate counts also provide useful evidence that the observed gains are not explained solely by generating more candidates.

Critical weaknesses

The methodological contribution appears limited to combining existing noise-space steering with group-relative policy gradients and an action-distance reward. The experiments do not sufficiently establish which parts of this combination are necessary. In particular, most benchmarks use only one optimizer step per sampled batch, so PPO clipping does not affect that update under the stated procedure. A comparison with a simpler normalized REINFORCE update would help establish what the GRPO formulation contributes beyond the existing steering framework.

The efficiency comparisons do not adequately account for generator computation. For example, the OGBench comparison matches the number of updates and states while DS-GRPO decodes 32 candidates per state. DSBC can instead precompute inversion labels and reuse them during training. Without systematic comparisons of total generator calls, training time, and memory, the practical efficiency advantage remains unclear.

The online comparisons involve differences that complicate attribution of the gains. DS-GRPO receives expert action labels, whereas DSRL learns from environment rewards, so equal environment-step budgets do not imply equal supervision. FlowDAgger is a more directly comparable baseline, but its inversion procedure is modified and unsuccessful inversion labels are discarded. The paper does not report the retained-label fraction or establish how this filtering affects the comparison.

The real-world results provide limited evidence of a reliable advantage over DSBC. The reported improvement corresponds to 16 versus 13 successful pick-and-place trials out of 30, and 8 versus 7 successful insertion trials out of 10. With these small evaluation budgets and no reported uncertainty estimates, the results support feasibility more strongly than consistent superiority over the baseline.

Questions for authors

What improvement does the GRPO formulation provide over normalized REINFORCE with the same candidates and reward? Given the single optimizer step used on most benchmarks, does removing PPO clipping change the results?

How do DS-GRPO and the inversion-based baselines compare under matched total computation, including label preprocessing and all generator calls? Please report training time and peak memory alongside success rates.

What fraction of FlowDAgger inversion labels is discarded on each task? How sensitive are its results to the filtering threshold and the modified inversion procedure?

How should the comparison with DSRL be interpreted given the difference in access to expert demonstrations? Are results available that control for the expert-information budget?

Can you report uncertainty estimates and configuration-level outcomes for the real-world comparisons? Do repeated training runs or additional evaluations support a consistent advantage over DSBC?

5 comments
AuthorsOpenReview ↗

Response to Reviewer VpMt [1/4]

We sincerely thank Reviewer VpMt for the detailed and technically precise review. We appreciate your recognition that DS-GRPO learns noise-space steering directly from action-space demonstration agreement, avoids sampler inversion, needs only forward access to the generator, and is evaluated across several backbones with matched-candidate comparisons. Your questions led to four new appendix sections (B.5, B.8, B.9 and B.10); all changes are highlighted in blue.

W1 / Q1: "What improvement does the GRPO formulation provide over normalized REINFORCE with the same candidates and reward? Given the single optimizer step used on most benchmarks, does removing PPO clipping change the results?"

TL;DR: You are right: with one Adam step per batch the clip is inactive, and the update is REINFORCE with a group-normalized advantage, as in the original GRPO. Removing the clip there gives bit-identical weights. The per-state group normalization appears to be what matters (with a step-size caveat below); the clip matters only when a batch is reused, which can speed up early learning but did not raise final success.

[New in revision] We now state this after Eq. (3.5), and App. B.9 ablates the update rule (3 seeds).

  1. Per-state vs. batch baseline. We replace the per-state normalization with one mean and standard deviation over all states and candidates of the batch, i.e. normalized REINFORCE with the same candidates and reward (Tab. 18):
Updatecube-doublecan
DS-GRPO (group baseline)31.8 ± 1.397.7 ± 0.8
Batch baseline (normalized REINFORCE)16.3 ± 0.987.0 ± 7.9

Success drops from 31.8 to 16.3% on cube-double and from 97.7 to 87.0% on can. The K candidates of a state share the state and the demonstration, so their mean score is a Monte Carlo estimate of that state's value, and the group advantage compares each candidate only with candidates for the same state; a batch baseline leaves differences in difficulty between states in the advantage. We do not claim this baseline as new: it is a standard variance-reduction device, and it is what makes a critic-free estimator work with demonstrations as action-space verifiers. Two caveats, now in B.9: the ablation changes centering and scaling together, and the batch baseline also moves the policy less per update, so part of the gap may come from a smaller effective step size (we did not re-tune its learning rate).

  1. Reusing a batch (E > 1), where the clip becomes active. On square we fixed the arms, seeds and analysis in writing before the runs and report all of them (Tabs. 19 and 20):
ArmE0.48M0.96M1.54M2.02MFinal (2.5M)
DS-GRPO157.2 ± 8.654.5 ± 1.252.8 ± 5.758.0 ± 11.065.7 ± 6.4
DS-GRPO, clip253.5 ± 4.355.2 ± 3.359.0 ± 7.958.7 ± 8.665.0 ± 7.8
DS-GRPO, clip460.0 ± 2.463.5 ± 4.159.8 ± 0.860.3 ± 13.359.7 ± 0.6
DS-GRPO, no clip454.7 ± 4.954.2 ± 9.652.8 ± 0.656.5 ± 10.667.5 ± 7.4

With the clip, E = 4 learns faster at first (63.5 vs. 54.5% at 0.96M steps), then stays near 60% while E = 1 keeps improving and ends higher; without the clip, E = 4 does not learn faster early but ends highest (67.5%). With three seeds, none of these differences is significant. We therefore keep one step per batch on OGBench, robomimic and FurnitureBench; LIBERO and the second-generation real-world tasks use two, which we have not ablated.

AuthorsOpenReview ↗

Response to Reviewer VpMt [2/4]

W2 / Q2: "How do DS-GRPO and the inversion-based baselines compare under matched total computation, including label preprocessing and all generator calls? Please report training time and peak memory alongside success rates."

TL;DR: We now report all of these (App. B.8). Offline, DS-GRPO makes more generator calls than DSBC, which we state in the Limitations; online, it makes fewer than FlowDAgger as we implement it. Compared with fine-tuning the backbone, it needs only forward access at inference-level memory, a property it shares with DSBC and DSRL. App. B.8 now states that this is what our efficiency claim refers to.

[New in revision] Measured with the training code of the main tables on one RTX 5090 (Tab. 16):

BenchmarkMethodGenerator calls (with backward)FLOPsTraining time (GPU-h)Peak memory (GiB)Success (%)
OGBench cube-double (NFE per run, 100k updates × 512 states)DSBC5.1×10⁸ (0), as run; 1.0×10⁷ if labels cached7.6×10¹⁴0.230.088.8
DS-BPTT1.6×10¹⁰ (1.6×10¹⁰)3.7×10¹⁶0.381.0919.4
DS-GRPO1.6×10¹⁰ (0)1.9×10¹⁶0.30–0.330.1628.2
robomimic square (decodes per 1,000 env steps, 2.5M steps)DSRL4.8×10⁵ (0)3.4×10¹⁷17.50.8096.5
FlowDAgger7,000 (5,000)5.2×10¹⁵2.750.2532.2
DS-BPTT2,810 (2,560)5.4×10¹⁴2.830.2653.5
DS-GRPO2,810 (0)3.2×10¹⁴3.99 as run (2.79 with one fused decode call)0.2568.2
  1. Offline, DS-GRPO costs more generator calls than DSBC. On OGBench it makes 32× the calls of DSBC as we ran it and about 1,650× those of DSBC with labels inverted once and cached (25× the FLOPs of DSBC as run); the training time is 0.30–0.33 vs. 0.23 GPU-h. As you suspected, DSBC's labels can be precomputed: our code re-inverted them per batch, which gives identical labels, and we corrected App. A.2. On LIBERO, DS-GRPO decodes 256,000 candidates per task against about 640 label inversions for DSBC.
  2. Matched total computation. Plotting success against cumulative generator calls on cube-double, including DSBC's label inversion, for K ∈ {2, …, 128} (Fig. 13, one seed), a larger K learns faster per iteration but not per call and does not raise the final success; at small K, DS-GRPO exceeds DSBC's final success after about 10× the calls of DSBC with cached labels, and at DSBC's as-run budget it reaches 18–27%.
  3. Online, DS-GRPO is cheaper than FlowDAgger as we implement it, in generator calls and FLOPs. On square it decodes 2,810 vs. 7,000 times per 1,000 environment steps, none with a backward pass (FlowDAgger: 5,000, from the refinement we use for the clamped sampler), at 16× fewer FLOPs; with the reference fixed-point inversion, which keeps about as many labels (Tab. 11), FlowDAgger would need fewer calls than DS-GRPO. As run, DS-GRPO's training time is 3.99 vs. 2.75 GPU-h for FlowDAgger; decoding all candidates of a step in one call, which gives the same decodes, brings DS-GRPO to 2.79.
  4. Peak memory. With π0.5, DS-GRPO trains at 7.2 GiB after torch.compile warm-up (13.1 GiB during warm-up), against 7.0 GiB at deployment; full fine-tuning with AdamW needs at least 27.9 GiB, and for StarVLA and Fast-WAM it does not fit on a 32 GB GPU (Tab. 17).

W3a / Q4: "How should the comparison with DSRL be interpreted given the difference in access to expert demonstrations?"

TL;DR: We agree that DSRL is not information-matched and now state this in the baseline description (Sec. 4.1). FlowDAgger is the supervision-matched baseline. We do not have results that control the expert-information budget.

We include DSRL as the representative noise-space steering method for online adaptation, learning from rewards rather than teacher labels. The two forms of supervision are complementary and can be combined, e.g. Flow Reversal Steering (Tang et al., 2026) adds a noise-space behavior-cloning term to the DSRL objective; we now name combining DS-GRPO with RL as future work in the Limitations.

AuthorsOpenReview ↗

Response to Reviewer VpMt [3/4]

W3b / Q3: "What fraction of FlowDAgger inversion labels is discarded on each task? How sensitive are its results to the filtering threshold and the modified inversion procedure?"

TL;DR: We report the kept fractions on all six tasks (App. B.5). Our inversion keeps about as many labels as the reference procedure run through the clamped sampler, and on square, loosening or removing the filter changes FlowDAgger's success by at most 5.5 points.

[New in revision] Why we modified the inversion. The DPPO sampler clamps its prediction to ±1 at every denoising step, so some expert actions have no preimage. Applied as is, the reference fixed-point solver diverges. Inverting the same teacher labels on fixed sets of 6,000 states per task (Tab. 11):

Inversion procedurecan: kept @10⁻³can: kept @10⁻¹square: kept @10⁻³square: kept @10⁻¹Median MSE (can / square)
Reference fixed-point, ignoring the clamp26.5%32.7%23.4%24.1%4.5×10² / 8.8×10¹⁶
Reference fixed-point, through the clamped chain93.1%98.9%72.7%93.2%2.0×10⁻⁴ / 3.4×10⁻⁴
Ours (actor-seeded + 20 Adam steps), clamped chain88.9%100.0%77.0%97.0%3.1×10⁻⁴ / 3.8×10⁻⁴

Ignoring the clamp, the decoded actions miss the expert action by a median MSE of 4×10² on can and 9×10¹⁶ on square. Through the clamped chain, our procedure (seeded at the actor's prediction, refined with 20 Adam steps) keeps about as many labels as the reference fixed-point iteration (88.9 vs. 93.1% on can, 77.0 vs. 72.7% on square).

Kept fractions in the runs of Tab. 3 (Tab. 10):

BenchmarkTaskKept (%)First 10% of trainingLast 10% of training
Robomimiclift90.5 ± 0.589.989.8
Robomimiccan90.2 ± 0.190.489.5
Robomimicsquare74.4 ± 0.577.174.1
FurnitureBenchone_leg69.6 ± 0.364.272.1
FurnitureBenchlamp18.4 ± 0.16.926.2
FurnitureBenchround_table61.3 ± 0.937.173.5

Sensitivity to the filter. On square, we re-ran FlowDAgger with the threshold raised to 10⁻² and 10⁻¹ and without any filter (Tab. 12):

Threshold on decoded-action MSEKept (%)Success (%)
10⁻³ (Tab. 3)74.4 ± 0.532.2 ± 2.1
10⁻²88.5 ± 1.035.5 ± 0.7
10⁻¹96.0 ± 0.332.8 ± 1.5
None10037.7 ± 1.6
DS-GRPO (Tab. 3)–68.2 ± 4.0

Without the filter FlowDAgger reaches 37.7%, still far below DS-GRPO's 68.2%. FlowDAgger also ends below the base policy on square, which we attribute to its deterministic noise policy behaving like the base policy at a small noise scale (Tab. 9, App. B.3). DS-GRPO needs no preimage: it scores whatever the sampler decodes, so it receives a learning signal at every visited state.

Reviewer_fiJXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DS-GRPO, a diffusion-steering method that adapts a frozen diffusion/flow policy by training a lightweight Gaussian policy over its initial noise.

Critical strengths
  1. The method is simple and needs only forward passes through the frozen sampler. That makes it applicable to black-box or non-invertible samplers, where inversion-based steering struggles.
  2. The evaluation is broad, showing the effectiveness of DS-GRPO.
Critical weaknesses
  1. The paper missed a baseline of backpropagating an action-space loss through the frozen sampler. Without this baseline, the gains cannot be attributed to the GRPO estimator rather than to simply using an action-space loss.
  2. Noise temperature is a confound. Lower-temperature sampling alone often improves diffusion policies, so a training-free “base policy with σ = 0.25 noise” baseline is needed.
Questions for authors
  1. How does DS-GRPO compare with directly backpropagating the action-space MSE through the frozen sampler, given that the FlowDAgger refinement already does this?
  2. What are the base-policy and teacher success rates on the Robomimic and FurnitureBench tasks? Why does DS-GRPO fall far behind DSRL on Square despite having expert labels?
4 comments
AuthorsOpenReview ↗

Response to Reviewer fiJX [1/2]

We sincerely thank Reviewer fiJX for the careful reading and the precise suggestions. We appreciate your recognition that DS-GRPO needs only forward passes through the frozen sampler, which makes it applicable where inversion-based steering struggles, and that the evaluation is broad. Both controls you asked for are now in the revision (App. B.4 and B.6), and the base and teacher rates are in Figs. 4–5 and App. B.3; all changes are highlighted in blue.

W1 / Q1: "The paper missed a baseline of backpropagating an action-space loss through the frozen sampler. Without this baseline, the gains cannot be attributed to the GRPO estimator rather than to simply using an action-space loss."

TL;DR: We added this baseline, DS-BPTT (App. B.6). It matches or exceeds the regression baselines, so the action-space loss explains part of the gain. It stays below DS-GRPO on the three state-based OGBench and the three robomimic tasks, clearly on cube-single and cube-double and within seed noise on the others, so the GRPO estimator, including its per-state normalization, adds to it. On LIBERO, the two reach comparable success, and DS-BPTT needs 2.2× the memory and 1.5× the time.

[New in revision] DS-BPTT shares the steering policy, σ schedule, states, K candidates per state, Adam steps per batch, frozen sampler, action-space distance, budget and evaluation with DS-GRPO. Candidates are drawn by reparameterization and the mean decoded-action distance is minimized by backpropagating through the whole denoising chain. The two methods therefore differ in the gradient estimator, which in DS-GRPO includes the per-state normalization of the scores (Eq. 3.4). Learning rates follow rules fixed before the runs (App. B.6):

BenchmarkTaskRegression baselineDS-BPTTDS-GRPO
OGBenchcube-single84.1 ± 3.7 (DSBC)83.6 ± 3.493.4 ± 1.9
OGBenchcube-double8.8 ± 1.2 (DSBC)19.4 ± 2.828.2 ± 2.4
OGBenchscene38.8 ± 2.4 (DSBC)39.0 ± 2.742.6 ± 2.0
Robomimiclift90.8 ± 3.7 (FlowDAgger)93.8 ± 1.595.7 ± 1.5
Robomimiccan72.3 ± 2.9 (FlowDAgger)95.2 ± 2.497.3 ± 0.8
Robomimicsquare32.2 ± 2.1 (FlowDAgger)53.5 ± 11.968.2 ± 4.0

DS-BPTT is below DS-GRPO by 3.6 to 9.8 points on OGBench and 1.9 to 14.7 points on robomimic, and matches or exceeds the regression baseline on every task. On robomimic the sampler clamps its prediction at every denoising step, which blocks part of the pathwise gradient, but the gap is largest on square, where the clamped fraction is moderate, so clamping alone does not explain it. On LIBERO with π0.5 (10 random tasks, 2 seeds, same hardware), DS-BPTT reaches 54.1 ± 0.5% and DS-GRPO 57.9 ± 1.1% (paired difference not significant), and the two differ mainly in cost:

MethodPeak GPU memory (GiB)Time per iteration (s)GPU-h per 2000-iteration run
DS-GRPO13.13.532.0
DS-BPTT, checkpointed per denoising step28.95.373.0
DS-BPTT, no checkpointingOOM (32 GB and 141 GB H200)––

Without activation checkpointing, DS-BPTT runs out of memory at our batch of 128 candidates even on a 141 GB H200. Its pathwise gradient also requires a differentiable sampler, whereas DS-GRPO applies unchanged to samplers that cannot be differentiated. On FlowDAgger: its refinement backpropagates the action distance through the sampler separately for each label, to refine an inverted noise that the steering policy then regresses on; DS-BPTT amortizes this optimization in the steering policy and is the direct pathwise counterpart of DS-GRPO.

AuthorsOpenReview ↗

Response to Reviewer fiJX [2/2]

W2: "Noise temperature is a confound. Lower-temperature sampling alone often improves diffusion policies, so a training-free 'base policy with σ = 0.25 noise' baseline is needed."

TL;DR: We added this training-free baseline on all ten OGBench, robomimic and FurnitureBench tasks (App. B.4, Tab. 9). A lower temperature helps the frozen base policy on some tasks, but stays far below DS-GRPO on every task, so it does not explain the gains.

[New in revision] The frozen base policy with initial noise from N(0, σ²I), σ ∈ {1, 0.5, 0.25}, under the paper's evaluation protocol:

BenchmarkTaskσ=1σ=0.5σ=0.25DS-GRPO
OGBenchcube-single10.48.03.693.4
OGBenchcube-double1.60.84.028.2
OGBenchscene2.42.00.042.6
OGBenchvisual-cube-single4.00.41.691.4
Robomimiclift75.535.512.595.7
Robomimiccan41.569.547.597.3
Robomimicsquare52.551.537.568.2
FurnitureBenchone_leg41.052.552.091.0
FurnitureBenchlamp3.22.72.961.9
FurnitureBenchround_table3.48.09.074.4

A lower temperature helps on can (41.5 → 69.5% at σ = 0.5), one_leg and round_table, and hurts on lift and square. On OGBench, where DS-GRPO's σ reaches its floor of 0.25 early in training, the base policy at σ = 0.25 succeeds in at most 4.0% of episodes. The gain therefore comes from the learned state- and goal-dependent steering policy rather than from narrower noise.

Q2a: "What are the base-policy and teacher success rates on the Robomimic and FurnitureBench tasks?"

[New in revision] Figs. 4 and 5 now show the base policy (dashed) and the teacher (dotted), and App. B.3 (Tab. 8) lists the rates:

BenchmarkTaskBaseTeacherDSRLFlowDAggerDS-GRPO
Robomimiclift75.5100.096.390.895.7
Robomimiccan41.599.595.772.397.3
Robomimicsquare52.591.596.532.268.2
FurnitureBenchone_leg41.097.137.761.091.0
FurnitureBenchlamp3.293.53.516.861.9
FurnitureBenchround_table3.496.41.729.674.4

Q2b: "Why does DS-GRPO fall far behind DSRL on Square despite having expert labels?"

TL;DR: DSRL optimizes the task reward and exceeds the teacher itself on square (96.5 vs. 91.5%), whereas DS-GRPO and FlowDAgger learn to reproduce the teacher's actions. Among the methods with the same supervision, DS-GRPO more than doubles FlowDAgger. We agree that its remaining gap to the teacher on square is not fully explained.

Expert labels give the teacher's behavior, not the task reward, so a demonstration-based method is not expected to exceed the teacher, while DSRL can. DS-GRPO's success on square rises slowly over the budget (window means of 52.9% in the first 0.5M steps and 62.2% in the last) and does not reach the teacher; we have not isolated the cause. FlowDAgger ends below the base policy on this task (32.2 vs. 52.5%), which we attribute to its deterministic noise policy behaving like the base policy at a small noise scale (37.5% at σ = 0.25, Tab. 9; App. B.3).

Summary: We thank the reviewer again for these two controls, which sharpen what the paper can claim: the action-space objective explains part of the gain, the GRPO estimator adds to it at lower cost, and a lower temperature does not account for it. We hope these additions address your concerns, and we would be grateful if you would consider them in your final assessment. We are happy to discuss any further questions.

As the rebuttal deadline approaches, we hope the reviewer has had a chance looking at our new experiments and that the requested experiments addressed their concerns.

Thank you for the response. The additional experiments addressed my main issues. Here I'm willing to increase my rating to 3.

Two points I would still like to see addressed in the final version. First, DS-BPTT recovers most of the gain over the regression baselines, and DS-GRPO is clearly better only on cube-single and cube-double; on LIBERO the difference is not significant. The main text should say this more directly. Second, DS-GRPO standardizes scores within each state while DS-BPTT does not, so the comparison does not fully isolate the estimator. On robomimic, the best DS-BPTT learning rate is also the largest one tried.

Reviewer_9RaHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper adapts a frozen diffusion or flow policy by learning a small policy over its initial noise. Instead of inverting demonstrations into noise targets, DS-GRPO samples several noises, decodes them, and scores the resulting action chunks by their squared distance to the demonstration. The experiments results are promising.

Critical strengths
  1. The motivation is easy to follow. If the goal is to match demonstrated actions, measuring the error after decoding makes sense.

  2. The evaluation covers a useful range of settings. The gains are not confined to one base model.

  3. The comparisons with AWR and best-of-K use the same number of candidates, helping show that the update rule contributes to the gains.

Critical weaknesses
  1. Training efficiency needs a clearer comparison. Matching updates or environment steps does not account for DS-GRPO decoding multiple candidates per state, while DSBC uses cached labels.

  2. Distance to a single demonstrated action does not necessarily reflect task success and may penalize other valid actions. Tests with multiple valid demonstrations per state, or an analysis linking action MSE to rollout success, would clarify this limitation.

  3. The real-world gains over DSBC amount to three additional successes in 30 pick-and-place trials and one in 10 insertion trials. More trials and varied initial configurations are needed to establish a reliable advantage.

Questions for authors

see weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer 9RaH [1/2]

We sincerely thank Reviewer 9RaH for the careful and constructive review. We are glad that you found the motivation easy to follow, the evaluation broad across base models, and the matched-candidate comparisons with AWR and best-of-K informative. Your comments led to three new analyses in the revision (App. B.7, B.8 and B.10); all changes are highlighted in blue. We address each point below.

W1: "Training efficiency needs a clearer comparison. Matching updates or environment steps does not account for DS-GRPO decoding multiple candidates per state, while DSBC uses cached labels."

TL;DR: We agree. We now report generator calls, FLOPs, training time and peak memory for all methods (App. B.8). Offline, DS-GRPO makes more generator calls than DSBC, which we now state in the Limitations; online, it makes fewer than FlowDAgger as we implement it. Compared with fine-tuning the backbone, it needs only forward access at inference-level memory, a property it shares with DSBC and DSRL. App. B.8 now states that this is what our efficiency claim refers to.

[New in revision] We measured each method with the training code of the main tables on one RTX 5090 (App. B.8, Tab. 16):

BenchmarkMethodGenerator calls (with backward)FLOPsTraining time (GPU-h)Peak memory (GiB)Success (%)
OGBench cube-double (NFE per run, 100k updates × 512 states)DSBC5.1×10⁸ (0), as run; 1.0×10⁷ if labels cached7.6×10¹⁴0.230.088.8
DS-BPTT1.6×10¹⁰ (1.6×10¹⁰)3.7×10¹⁶0.381.0919.4
DS-GRPO1.6×10¹⁰ (0)1.9×10¹⁶0.30–0.330.1628.2
robomimic square (decodes per 1,000 env steps, 2.5M steps)DSRL4.8×10⁵ (0)3.4×10¹⁷17.50.8096.5
FlowDAgger7,000 (5,000)5.2×10¹⁵2.750.2532.2
DS-BPTT2,810 (2,560)5.4×10¹⁴2.830.2653.5
DS-GRPO2,810 (0)3.2×10¹⁴3.99 as run (2.79 with one fused decode call)0.2568.2
  1. Offline (OGBench), DS-GRPO costs more generator calls than DSBC. It makes 32× the calls of DSBC as we ran it, and about 1,650× the calls of DSBC with labels inverted once and cached; its FLOPs are 25× those of DSBC as run. Because the generator is small, the training time is close (0.30–0.33 vs. 0.23 GPU-h). Our DSBC code re-inverted the labels of each batch; the inversion is deterministic, so caching gives identical labels, and we corrected the description in App. A.2.
  2. Much of the K = 32 budget is not needed. Plotting success against cumulative generator calls on cube-double for K ∈ {2, …, 128} (Fig. 13, one seed), a larger K learns faster per iteration but not per call, and does not raise the final success. At small K, DS-GRPO exceeds DSBC's final success after about 10× the calls of DSBC with cached labels.
  3. Online (robomimic square), DS-GRPO makes fewer generator calls than FlowDAgger as we implement it: 2,810 vs. 7,000 per 1,000 environment steps, none of them with a backward pass (FlowDAgger: 5,000, from the refined inversion we use for the clamped sampler), at 16× fewer FLOPs. With the reference fixed-point inversion, which keeps about as many labels, FlowDAgger would need fewer calls than DS-GRPO. As run, DS-GRPO's training time is 3.99 vs. 2.75 GPU-h for FlowDAgger; decoding the candidates of all states in one call, which gives the same decodes, brings DS-GRPO to 2.79, since the simulator takes most of the time.
  4. Memory, compared with fine-tuning the backbone. DS-GRPO needs only forward passes of the base policy: with π0.5 it trains at 7.2 GiB after torch.compile warm-up (13.1 GiB during warm-up), against 7.0 GiB at deployment and at least 27.9 GiB for full fine-tuning with AdamW (Tab. 17).
AuthorsOpenReview ↗

Response to Reviewer 9RaH [2/2]

W2: "Distance to a single demonstrated action does not necessarily reflect task success and may penalize other valid actions. Tests with multiple valid demonstrations per state, or an analysis linking action MSE to rollout success, would clarify this limitation."

TL;DR: We believe that this distance is a proxy for success, an assumption that also underlies regression-based behavior cloning and DAgger-style methods. We added an analysis linking held-out action MSE to rollout success (App. B.7): MSE orders the methods consistently with success, but it does not resolve differences within a method. We now state the limitation explicitly.

[New in revision] For each of the three state-based OGBench tasks we took 13 policies: the frozen prior, DS-GRPO at 80%, 90% and 100% of training for three seeds, and one retrained DSBC seed at the same three points. Each was evaluated once under the protocol of Tab. 7 (so the values differ slightly from Tab. 2), with held-out MSE on 20,000 validation queries (App. B.7, Fig. 12; MSE ×10⁻² / success %, mean per method):

TaskFrozen prior: MSE / SRDSBC: MSE / SRDS-GRPO: MSE / SRSpearman ρ [95% CI]
cube-single5.30 / 10.03.49 / 81.52.37 / 93.40.64 [−0.01, 0.94]
cube-double5.90 / 1.64.47 / 11.33.08 / 27.80.54 [−0.16, 0.93]
scene6.76 / 5.25.18 / 38.13.49 / 42.60.54 [−0.09, 0.94]

Between methods, lower MSE goes with higher success on all three tasks. Within a method, MSE varies by less than 1% across checkpoints and does not resolve their differences in success, so the rank correlation is positive but its confidence interval includes zero. DS-GRPO also minimizes this distance directly, so its low MSE is partly by construction. We therefore read action MSE as a proxy that orders these methods, not as a substitute for success. As in regression-based behavior cloning, a squared distance can favor actions near the mean of nearby demonstrations; in DS-GRPO it enters only through the reward, so a verifier over several valid demonstrations, or a task reward, can replace it without changing the estimator. We added this to the Limitations and leave the multi-demonstration test to future work.

W3: "The real-world gains over DSBC amount to three additional successes in 30 pick-and-place trials and one in 10 insertion trials. More trials and varied initial configurations are needed to establish a reliable advantage."

TL;DR: We agree that the original margins are within noise. We now report confidence intervals and exact tests, removed the "clear margin" claim, and added two tasks with second-generation Wuji hands (30 trials per method), on which DS-GRPO again has the highest success.

[New in revision] (App. B.10, Tab. 21.) On the original tasks, the differences to DSBC are not significant (p = 0.61 and 1.00). On Place-Plate and Screw-Lightbulb, the base policy is π0.5 fine-tuned on 200 demonstrations, and DSBC and DS-GRPO adapt it offline from 50 new demonstrations:

Task (trials per method)BaseDSBCDS-GRPOp vs DSBCp vs base
Pick-and-Place (30)11 (36.7%) [21.9, 54.5]13 (43.3%) [27.4, 60.8]16 (53.3%) [36.1, 69.8]0.610.30
Peg-Insertion (10)2 (20.0%) [5.7, 51.0]7 (70.0%) [39.7, 89.2]8 (80.0%) [49.0, 94.3]1.000.023
Place-Plate, new (30)6 (20.0%) [9.5, 37.3]9 (30.0%) [16.7, 47.9]17 (56.7%) [39.2, 72.6]0.0670.007
Screw-Lightbulb, new (30)3 (10.0%) [3.5, 25.6]4 (13.3%) [5.3, 29.7]9 (30.0%) [16.7, 47.9]0.210.10

DS-GRPO reaches 17/30 and 9/30 against 9/30 and 4/30 for DSBC, while on these two tasks DSBC stays close to the base policy. Per task, the difference to DSBC is not significant (p = 0.067 and 0.21); pooled over the two tasks with an exact test stratified by task, which we did not fix before the evaluation, it is significant (p = 0.016). Each steering policy was trained once. Each new task is evaluated from one fixed object placement, with the trials of each method run as one block, and success is defined strictly (the plate stands stably in the rack; the bulb stays lit after release; App. B.10). The new tasks therefore test whether the advantage holds across tasks and hardware, rather than across initial configurations, which remains a limitation of our real-world evaluation. The direction of the difference is the same on all four tasks, while the margin on any single task remains within the uncertainty of 10 to 30 trials.

Summary: We thank the reviewer again for these suggestions, which made the cost, the proxy and the real-world evidence of the paper much more transparent. We hope the new analyses address your concerns, and we would be grateful if you would take them into account in your final assessment. We are happy to answer any further questions.

As the rebuttal deadline approaches, we hope the reviewer has had a chance looking at our new experiments and that the requested experiments addressed their concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revision

We sincerely thank all reviewers and the AC for their time and constructive feedback. We are encouraged that the reviewers found the motivation clear (9RaH), the method simple and applicable to samplers that cannot be inverted or differentiated (fiJX, VpMt), and the evaluation broad across backbones and adaptation settings (9RaH, fiJX, VpMt), with informative matched-candidate comparisons (9RaH, VpMt).

DS-GRPO adapts a frozen generative policy by steering its initial noise, using demonstrations as action-space verifiers and group-relative policy gradients, without inversion, a critic, or backpropagation through the sampler. We took the reviewers' concerns seriously and made an extensive effort to address them: the revision adds ten new sets of experiments, nine in simulation and one on real robots with two entirely new tasks on second-generation dexterous hands, presented in eight new appendix sections, 14 new tables and 3 new figures. The main text stays within 10 pages, and all updates are highlighted in blue. Major updates:

  1. Compute and memory accounting (9RaH-W1, VpMt-W2): generator calls, FLOPs, training time and peak memory for all methods, success versus generator calls for K ∈ {2, …, 128}, and peak memory on three VLA/WAM backbones (App. B.8, Tabs. 16–17, Fig. 13). Offline, DS-GRPO makes more generator calls than DSBC, which we now state in the Limitations; online, it makes fewer than FlowDAgger as we implement it; compared with fine-tuning the backbone, it needs only forward access at inference-level memory.
  2. Additional Ablations regarding GRPO-related Components (VpMt-W1): with one Adam step per batch the update is REINFORCE with a group-normalized advantage (Sec. 3.2); replacing the per-state normalization with a batch baseline lowers success substantially; reusing a batch for clipped steps can speed up early learning (not significant with three seeds) but does not raise final success, in a pre-registered study (App. B.9, Tabs. 18–20).
  3. Pathwise-gradient baseline DS-BPTT (fiJX-W1): below DS-GRPO on the three state-based OGBench and three robomimic tasks, comparable on LIBERO at 2.2× the memory (App. B.6, Tabs. 13–15).
  4. Training-free temperature baseline (fiJX-W2): a lower noise scale stays far below DS-GRPO on all ten OGBench, robomimic and FurnitureBench tasks (App. B.4, Tab. 9).
  5. Base and teacher success rates (fiJX-Q2a): reference lines in Figs. 4–5 and App. B.3.
  6. FlowDAgger label retention and filter sensitivity (VpMt-W3b): kept fractions on all tasks, inversion procedures, threshold sweep (App. B.5, Tabs. 10–12).
  7. Action MSE versus success (9RaH-W2): App. B.7, Fig. 12, and an explicit limitation.
  8. Real-world uncertainty and two new tasks (9RaH-W3, VpMt-W4): confidence intervals and exact tests for all comparisons, and Place-Plate and Screw-Lightbulb with second-generation Wuji hands (Tab. 4, App. B.10, Tab. 21).
  9. Clarifications: DSRL is not information-matched (Sec. 4.1); three descriptions of the setup that did not match our code were corrected (DSBC's per-batch label inversion on OGBench and FlowDAgger's collection schedule in App. A.2; the chunk-steps per collection on square in Tab. 6), with no change to the reported results.

In summary, the reviewers' questions led us to measure what DS-GRPO costs, to isolate which parts of the method matter, to test it against the strongest alternative explanations (a pathwise gradient, a lower noise temperature, the FlowDAgger filter), and to strengthen the real-world evidence. We have made every effort to address all concerns raised by the reviewers. We believe the revised paper gives a significantly clearer and better-supported account of when and why steering with action-space verification works, and we are grateful to the reviewers and the AC for helping us get there. We remain happy to answer any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.