- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies DSRL which improves a frozen diffusion or flow policy by learning a policy over its input noise and shows with a noise probe that the frozen policy carries atypical noise into poor actions more so as the noise dimension grows. It proposes Typical DSRL whose latent policy is the prior shifted by a learned state dependent mean so its KL divergence to the prior is exactly half the squared shift and this KL is bounded by a budget stated in shell widths. Experiments on eleven robomimic, Adroit and Gym tasks which shows Typical DSRL achieving the best or tied best result on eight tasks while training several times faster than the strongest competing methods.
Critical strengths
- The noise probe is a usefu diagnosis as it shows clearly that the frozen policy turns unusual noise into bad actions and that this gets worse as the noise dimension grows. The fix is simple since keeping the prior's noise and only shifting its mean gives an exact KL to the prior and the same budget also limits how far the actions move from the frozen policy.
- The experiments are thorough and fair. The method performs strongly across eleven tasks at a fraction of the training cost of the best baselines all methods are run in the same training loop with the same budgets and the paper honestly reports the cases that go against it.
Critical weaknesses
- It is not clear to me that typicality is what drives the gains. The no constraint run is DSRL-SAC which also changes the actor and the temperature as per appendix and under the same stochastic evaluation LP-DS-SAC whose shift is bounded but in raw units has a higher mean on on four of eight tasks (like 0.94 vs 0.82 on door). The paper says part of the lead comes from deploying the mean latent but it looks to me that Typical DSRL pulls ahead of these baselines mainly when it removes the noise at deployment.
- The shell width budget which i feel is the main difference from LP-DS is never compared with simpler budgets on the same actor and is not swept on the high-dimensional Adroit tasks. Alternatively if authors can show Typical DSRL trained without the budget (same actor, fixed ) and an sweep on one Adroit task that would be helpful.
Questions for authors
- LP-DS and LP-DS-SAC are reported only stochastically since they have no mean policy. Could their existing checkpoints also be evaluated without noise to show how much of the lead comes from noise free deployment?
- How were the per task values of chosen for the ten tasks other than square?
3 comments
**Response to Reviewer yLpA (2/2)**
W1, Q1: LP-DS and LP-DS-SAC without noise
We agree with the observation. Under stochastic evaluation LP-DS-SAC has the higher mean on four of the eight success-rate tasks (lift, door, hammer and relocate), and on these tasks Typical DSRL is ahead only with its maximum-likelihood deployment.
As the reviewer asks, we evaluated the existing checkpoints of LP-DS and LP-DS-SAC without noise: the input noise is set to zero and each method applies its learned correction to the zero latent. Every cell evaluates the final checkpoints on 200 episodes (transport 100) and the same initial states for every method.
| Method | Task | Method as deployed | Method at zero noise | Ours |
|---|---|---|---|---|
| LP-DS-SAC | lift | 0.730 | 0.367 | 0.905 |
| transport | 0.380 | 0.015 | 0.690 | |
| door | 0.873 | 0.805 | 0.998 | |
| hammer | 0.818 | 0.995 | 0.965 | |
| pen | 0.895 | 0.760 | 0.992 | |
| relocate | 0.820 | 0.660 | 0.992 | |
| LP-DS | lift | 0.985 | 0.893 | 0.905 |
| can | 0.988 | 0.988 | 1.000 | |
| square | 0.863 | 0.752 | 0.935 | |
| door | 0.950 | 0.975 | 0.998 | |
| hopper | 2368 | 2986 | 3165 | |
| walker | 3682 | 3839 | 3941 |
- LP-DS-SAC. Zero noise lowers LP-DS-SAC on five of its six tasks, so its gap to Typical DSRL widens there. It raises LP-DS-SAC on hammer, where LP-DS-SAC is then above Typical DSRL.
- LP-DS. Zero noise raises LP-DS on door, hopper and walker, where its gap to Typical DSRL narrows. It leaves can unchanged and lowers lift and square. On lift, LP-DS is above Typical DSRL as deployed and no longer above it at zero noise.
- Both. With the noise removed from the baselines, Typical DSRL is at or above them on 11 of the 12 rows. The exception is hammer against LP-DS-SAC.
Removing the noise is therefore not a gain by itself. It lowers LP-DS-SAC on five of six tasks and LP-DS on two of six, and the Typical DSRL actor trained without the budget scores 0.000 under the same deployment (first point above). The gain depends on the actor: Typical DSRL learns the best latent action at the given state, whereas LP-DS and LP-DS-SAC learn a shift from the sampled noise.
Q2: choice of α
Square is the only task on which Typical DSRL was trained at more than one value of α (Fig. 9); every other task was trained at its value alone. To test whether the per-task values matter, we re-ran Typical DSRL at α = 0.05 on the five tasks whose α is another value:
| Task | paper's α | Ours, paper's α | Ours, α = 0.05 |
|---|---|---|---|
| lift | 0 | 0.883 ± 0.067 | 0.985 ± 0.005 |
| hammer | 0.1 | 0.968 ± 0.005 | 0.977 ± 0.010 |
| walker | 0.1 | 3925 ± 17 | 4003 ± 37 |
| hopper | 0 | 3175 ± 25 | 3171 ± 6 |
| halfcheetah | 0 | 5015 ± 2 | 4973 ± 4 |
α = 0.05 is higher on lift and walker, the same within the seed spread on hammer and hopper, and 0.8% lower on halfcheetah. The other six tasks already use α = 0.05, so one value serves all eleven tasks. We also ran this single setting (η = 0.5, α = 0.05) on seeds 1–5 on all eleven tasks (1–4 on transport). The five-seed means are within 0.01 of the two-seed means on every success-rate task except transport (0.695 vs 0.647), and within 0.5% on Gym.
Thank you for the detailed response and the additional experiments. The run without the budget on the same actor and the comparison with simpler budgets and the sweep on Adroit address my concerns and support the role of the shell width budget. The zero noise evaluation also shows that removing the noise alone does not explain the lead. I maintain my score as the method matches rather than surpasses the strongest baseline on several tasks. I encourage the authors to add these results and the five seed runs to the paper.
**Response to Reviewer yLpA (1/2)**
We thank the reviewer for the constructive and encouraging feedback and recognition of our noise probe as a useful diagnosis, the simplicity of our method, and the thoroughness and fairness of our experiments. We carefully address each concern below with the suggested ablations and evaluations.
W1, W2: the budget on the same actor
We agree that DSRL-SAC is not a clean no-constraint run, since it also changes the actor and the temperature. We ran the comparisons the reviewer suggests on the Typical DSRL actor itself.
Same actor without the budget. We trained Typical DSRL with only the KL budget removed: the same mean-shift actor with σ = 1, the same fixed α, the same TD critic and the same deployment. It scores 0.000 on seven of the eight success-rate tasks (all except pen), −8 on walker and 826 on halfcheetah, and is below the frozen pretrained policy on all eleven tasks.
η on Adroit. On the same actor we swept the budget width η on pen and relocate, with hopper as a low-dimensional reference:
On both Adroit tasks the score is highest at the paper's η = 0.5 and changes little for η ≤ 1. It falls steadily as the budget widens beyond η = 1 and ends at the run without the budget.
Simpler rules in place of the √(2d) budget. On the same actor we trained two simpler rules. The first is a budget proportional to d, which is a fixed mean squared shift per coordinate, a bound in raw units. It is set equal to ours at d = 28 (lift, can, square), so the two differ only away from it. The second is a KL penalty with its weight fixed at 1 and no budget.
The three rules are within 1% of each other at d = 12. At d = 192 and 240 both simpler rules are below the √(2d) budget. The proportional budget is 2.6 and 2.9 times wider than ours there, and the fixed weight lets the KL grow to 206 and 50 nats against budgets of about 10.