acceptodds
Under review as a conference paper at ICLR 2027

Typicality Constrained Latent Reinforcement Learning for Generative Policies

Abstract

Diffusion steering via reinforcement learning (DSRL) improves a frozen diffusion or flow-matching policy by learning a latent policy over its input noise. The frozen policy, however, acts well only on noise typical of its Gaussian prior, with a tolerance that shrinks as the dimension grows. Existing steering methods bound the noise in raw units re-tuned per task and need a costly critic distilled through the frozen policy. We propose Typical DSRL, whose latent policy is the prior shifted by a learned mean, so its KL divergence from the prior is exact; bounding it in widths of the prior's typical shell lets one budget serve every dimension. Because its critic sees only typical noise, it needs no distillation. On eleven robomimic, Adroit and Gym tasks, with one budget, Typical DSRL is Pareto-optimal in performance against training time; baselines that come close train two to twenty-four times longer. Typicality is needed only in training: deploying the latent policy's mean, which lies far from the typical shell, outperforms sampling on every task.

Primary area:
reinforcement learning
Keywords:
diffusion steering, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yLpAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies DSRL which improves a frozen diffusion or flow policy by learning a policy over its input noise and shows with a noise probe that the frozen policy carries atypical noise into poor actions more so as the noise dimension grows. It proposes Typical DSRL whose latent policy is the prior shifted by a learned state dependent mean so its KL divergence to the prior is exactly half the squared shift and this KL is bounded by a budget stated in shell widths. Experiments on eleven robomimic, Adroit and Gym tasks which shows Typical DSRL achieving the best or tied best result on eight tasks while training several times faster than the strongest competing methods.

Critical strengths
  1. The noise probe is a usefu diagnosis as it shows clearly that the frozen policy turns unusual noise into bad actions and that this gets worse as the noise dimension grows. The fix is simple since keeping the prior's noise and only shifting its mean gives an exact KL to the prior and the same budget also limits how far the actions move from the frozen policy.
  2. The experiments are thorough and fair. The method performs strongly across eleven tasks at a fraction of the training cost of the best baselines all methods are run in the same training loop with the same budgets and the paper honestly reports the cases that go against it.
Critical weaknesses
  1. It is not clear to me that typicality is what drives the gains. The no constraint run is DSRL-SAC which also changes the actor and the temperature as per appendix and under the same stochastic evaluation LP-DS-SAC whose shift is bounded but in raw units has a higher mean on on four of eight tasks (like 0.94 vs 0.82 on door). The paper says part of the lead comes from deploying the mean latent but it looks to me that Typical DSRL pulls ahead of these baselines mainly when it removes the noise at deployment.
  2. The shell width budget which i feel is the main difference from LP-DS is never compared with simpler budgets on the same actor and η\eta is not swept on the high-dimensional Adroit tasks. Alternatively if authors can show Typical DSRL trained without the budget (same actor, fixed α\alpha) and an η\eta sweep on one Adroit task that would be helpful.
Questions for authors
  1. LP-DS and LP-DS-SAC are reported only stochastically since they have no mean policy. Could their existing checkpoints also be evaluated without noise to show how much of the lead comes from noise free deployment?
  2. How were the per task values of α\alpha chosen for the ten tasks other than square?
3 comments
AuthorsOpenReview ↗

**Response to Reviewer yLpA (1/2)**

We thank the reviewer for the constructive and encouraging feedback and recognition of our noise probe as a useful diagnosis, the simplicity of our method, and the thoroughness and fairness of our experiments. We carefully address each concern below with the suggested ablations and evaluations.

W1, W2: the budget on the same actor

We agree that DSRL-SAC is not a clean no-constraint run, since it also changes the actor and the temperature. We ran the comparisons the reviewer suggests on the Typical DSRL actor itself.

Same actor without the budget. We trained Typical DSRL with only the KL budget removed: the same mean-shift actor with σ = 1, the same fixed α, the same TD critic and the same deployment. It scores 0.000 on seven of the eight success-rate tasks (all except pen), −8 on walker and 826 on halfcheetah, and is below the frozen pretrained policy on all eleven tasks.

η on Adroit. On the same actor we swept the budget width η on pen and relocate, with hopper as a low-dimensional reference:

ηhopper (d = 12)pen (d = 192)relocate (d = 240)
0.12531700.9370.990
0.2531480.9680.995
0.5 (paper)31710.9880.997
131660.9750.947
218850.8570.612
411180.6870.128
no budget1830.0920.000

On both Adroit tasks the score is highest at the paper's η = 0.5 and changes little for η ≤ 1. It falls steadily as the budget widens beyond η = 1 and ends at the run without the budget.

Simpler rules in place of the √(2d) budget. On the same actor we trained two simpler rules. The first is a budget proportional to d, which is a fixed mean squared shift per coordinate, a bound in raw units. It is set equal to ours at d = 28 (lift, can, square), so the two differ only away from it. The second is a KL penalty with its weight fixed at 1 and no budget.

Taskd√(2d) budget (ours)budget ∝ dKL weight fixed at 1
hopper123171 ± 63167 ± 83147 ± 4
pen1920.988 ± 0.0020.962 ± 0.0050.543 ± 0.050
relocate2400.997 ± 0.0000.693 ± 0.0030.340 ± 0.067

The three rules are within 1% of each other at d = 12. At d = 192 and 240 both simpler rules are below the √(2d) budget. The proportional budget is 2.6 and 2.9 times wider than ours there, and the fixed weight lets the KL grow to 206 and 50 nats against budgets of about 10.

AuthorsOpenReview ↗

**Response to Reviewer yLpA (2/2)**

W1, Q1: LP-DS and LP-DS-SAC without noise

We agree with the observation. Under stochastic evaluation LP-DS-SAC has the higher mean on four of the eight success-rate tasks (lift, door, hammer and relocate), and on these tasks Typical DSRL is ahead only with its maximum-likelihood deployment.

As the reviewer asks, we evaluated the existing checkpoints of LP-DS and LP-DS-SAC without noise: the input noise is set to zero and each method applies its learned correction to the zero latent. Every cell evaluates the final checkpoints on 200 episodes (transport 100) and the same initial states for every method.

MethodTaskMethod as deployedMethod at zero noiseOurs
LP-DS-SAClift0.7300.3670.905
transport0.3800.0150.690
door0.8730.8050.998
hammer0.8180.9950.965
pen0.8950.7600.992
relocate0.8200.6600.992
LP-DSlift0.9850.8930.905
can0.9880.9881.000
square0.8630.7520.935
door0.9500.9750.998
hopper236829863165
walker368238393941
  • LP-DS-SAC. Zero noise lowers LP-DS-SAC on five of its six tasks, so its gap to Typical DSRL widens there. It raises LP-DS-SAC on hammer, where LP-DS-SAC is then above Typical DSRL.
  • LP-DS. Zero noise raises LP-DS on door, hopper and walker, where its gap to Typical DSRL narrows. It leaves can unchanged and lowers lift and square. On lift, LP-DS is above Typical DSRL as deployed and no longer above it at zero noise.
  • Both. With the noise removed from the baselines, Typical DSRL is at or above them on 11 of the 12 rows. The exception is hammer against LP-DS-SAC.

Removing the noise is therefore not a gain by itself. It lowers LP-DS-SAC on five of six tasks and LP-DS on two of six, and the Typical DSRL actor trained without the budget scores 0.000 under the same deployment (first point above). The gain depends on the actor: Typical DSRL learns the best latent action at the given state, whereas LP-DS and LP-DS-SAC learn a shift from the sampled noise.

Q2: choice of α

Square is the only task on which Typical DSRL was trained at more than one value of α (Fig. 9); every other task was trained at its value alone. To test whether the per-task values matter, we re-ran Typical DSRL at α = 0.05 on the five tasks whose α is another value:

Taskpaper's αOurs, paper's αOurs, α = 0.05
lift00.883 ± 0.0670.985 ± 0.005
hammer0.10.968 ± 0.0050.977 ± 0.010
walker0.13925 ± 174003 ± 37
hopper03175 ± 253171 ± 6
halfcheetah05015 ± 24973 ± 4

α = 0.05 is higher on lift and walker, the same within the seed spread on hammer and hopper, and 0.8% lower on halfcheetah. The other six tasks already use α = 0.05, so one value serves all eleven tasks. We also ran this single setting (η = 0.5, α = 0.05) on seeds 1–5 on all eleven tasks (1–4 on transport). The five-seed means are within 0.01 of the two-seed means on every success-rate task except transport (0.695 vs 0.647), and within 0.5% on Gym.

Thank you for the detailed response and the additional experiments. The run without the budget on the same actor and the comparison with simpler budgets and the η\eta sweep on Adroit address my concerns and support the role of the shell width budget. The zero noise evaluation also shows that removing the noise alone does not explain the lead. I maintain my score as the method matches rather than surpasses the strongest baseline on several tasks. I encourage the authors to add these results and the five seed runs to the paper.

Reviewer_1tnrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem of improving the performance of the generative policy through RL given a pretrained diffusion model or flow matching model. The authors proposed Typical DSRL, which adjusts the generated action distribution by learning the mean of the initial Gaussian noise. At the same time, a KL divergence-based constraint is also introduced to prevent the learned noise distribution from deviating too much from the original standard Gaussian distribution. In addition, Typical DSRL uses constrained SAC to directly train the latent space TD critic, avoiding the significant computational overhead of critic distillation. The authors conducted experiments on 11 robotic control tasks, and the results show that this method can achieve comparable or better performance to existing methods with lower training costs.

Critical strengths
  1. The method in the paper is very concise, that is, improving the generative policy by learning the mean of the initial Gaussian noise through the pretrained model. And it don't need to update the original model. At the same time, the KL constraint itself is easy to compute and does not require additional approximations.
  2. The experimental results are good. The author conducted experiments on 11 robot control tasks, and compared to various baselines, Typical DSRL was able to achieve comparable or better performance with lower training costs, especially by directly using the TD critic, avoiding the computational overhead brought by critic distillation, which is commendable.
  3. The paper also provides a relatively comprehensive experimental analysis, such as the impact of different initial noise on action generation and the effect of the KL budget parameter η\eta on performance. These analyzes contribute to a better understanding of the mechanisms of latent-space policy optimization.
Critical weaknesses
  1. The novelty is limited. Optimizing the initial noise of the diffusion model through RL has already been studied in works, such as DSRL [1], FIND [3], and Noise PPO [4]. In paricular, Noise PPO [4] also learns a conditional Gaussian noise policy with KL regularization. LP-DS [2] has also introduced constraints on noise perturbation. The main difference in this paper lies in the use of a fixed covariance, which allows for accurate calculation of the KL divergence. These designs are primarily based on the properties of existing Gaussian distribution and trust-region, and I believe the paper needs to further explain its unique contributions.
  2. The paper compares Typical DSRL with DSRL-SAC, but during the comparison, not only is the KL constraint different, but the parameterization method of the latent policy, covariance, etc., are also changed. Therefore, the current performance improvement cannot be directly attributed to the typicality constraint. I think the authors should compare the cases with and without the KL constraint under the same Gaussian actor and SAC setting.
  3. I'm confused about the specific advantages of latent mean shift compared to action residual. The authors mentioned in Section 4.2 that the frozen flow-matching decoder on the Adroit task is almost an affine mapping, if so, performing a mean shift on the initial noise is approximately equivalent to adding a residual to the final action. Although the paper compares DICE-RL, the two methods have significant differences, making it difficult to determine whether the latent mean shift itself has unique advantages.
Questions for authors
  1. What specific role does the typicality constraint play in improving TD critic learning? The paper suggests that maintaining the typicality of the latent can make the TD critic more reliable, but it is still unclear where this improvement comes from. Can the author compare the cases with and without KL constraint under the same Gaussian actor setup, and further analyze the accuracy of the TD critic?
  2. What unique advantages does latent mean shift have compared to action-space residual correction? See weakness.
  3. Why does training on typical latents yield a better-performing atypical mean latent? The article emphasizes the importance of typicality during the training phase, but the learned mean used during testing is outside the typical shell and performs better than stochastic sampling. Although the authors attribute the performance improvement to the removal of action noise, I still don't quite understand how the typicality constraint helps in learning a better deterministic policy. Could the author further explain this mechanism?
4 comments
AuthorsOpenReview ↗

**Response to Reviewer 1tnr (1/3)**

We thank the reviewer for the detailed feedback and recognition of the conciseness of our method, its performance at a lower training cost without critic distillation, and our experimental analysis. We carefully address each concern below with additional ablations, analyses, and clarifications.

W1: differences from Noise PPO and LP-DS

We agree that steering the initial noise with RL is not new, that Noise PPO regularizes a Gaussian noise policy with a KL term, and that LP-DS constrains a noise perturbation. Typical DSRL differs in the constraint: the exact KL of the latent policy to the prior is held at a budget in widths of the prior's shell, η√(2d), so that η = 0.5 serves all eleven tasks, from d = 12 to d = 240.

  • Noise PPO adds the KL to the loss as a penalty with a constant coefficient. A penalty fixes the price of leaving the prior and not the distance, so how far the policy moves depends on the task. Typical DSRL fixes the distance, with a Lagrange multiplier that holds the KL at the budget. Noise PPO also learns the variance, where we fix the covariance at the prior's so that the whole budget goes to the mean. And it treats image generation as a one-step problem with no critic over the noise, where in control the effect of a latent returns through the state, so our actor follows a TD critic over latents.
  • LP-DS bounds the mean squared perturbation per coordinate, a threshold in raw units that is set per environment. A tolerance per coordinate does not transfer across tasks, since the same inflation of the latent norm costs more as d grows (Sec. 3.1). Its perturbation depends on the sampled noise, w = ε + Δθ(s, ε), so its bound only approximates the KL and it has no closed-form mean latent to deploy. Our shift depends on the state only, w = Δθ(s) + ε: half its squared norm is the KL exactly, and Δθ(s) is the latent that we deploy. LP-DS also distills its latent critic from an action critic and trains 2.6 to 7 times longer than Typical DSRL (Table 10), which uses a TD critic.

We tested both forms on the Typical DSRL actor, with nothing else changed (all runs at α = 0.05). LP-DS's form is a budget proportional to d, set equal to ours at d = 28. Noise PPO's form is a KL penalty with its weight fixed at 1.

Taskd√(2d) budget (ours)budget ∝ dKL weight fixed at 1
hopper123171 ± 63167 ± 83147 ± 4
pen1920.988 ± 0.0020.962 ± 0.0050.543 ± 0.050
relocate2400.997 ± 0.0000.693 ± 0.0030.340 ± 0.067

At d = 12 the three rules are within 1% of each other. At d = 192 and 240 both other forms are below the √(2d) budget.

W2: the same actor with and without the KL constraint

We agree that DSRL-SAC also changes the actor and the temperature. We trained Typical DSRL with only the KL constraint removed: the same Gaussian mean-shift actor with σ = 1, the same fixed α, the same SAC training with a TD critic, and the same deployment.

TaskPretrained policy (frozen)Same actor, no KL constraintOurs
lift0.718 ± 0.0180.000 ± 0.0000.883 ± 0.067
can0.545 ± 0.0350.000 ± 0.0000.998 ± 0.003
square0.520 ± 0.0300.000 ± 0.0000.920 ± 0.027
transport0.180 ± 0.0200.000 ± 0.0000.647 ± 0.052
door0.665 ± 0.0150.000 ± 0.0000.992 ± 0.002
hammer0.430 ± 0.0100.000 ± 0.0000.968 ± 0.005
pen0.775 ± 0.0050.092 ± 0.0080.988 ± 0.002
relocate0.295 ± 0.0150.000 ± 0.0000.997 ± 0.000
hopper1637 ± 17183 ± 593175 ± 25
walker2747 ± 12−8 ± 43925 ± 17
halfcheetah4102 ± 12826 ± 2515015 ± 2

Without the constraint, the same actor ends below the pretrained policy on all eleven tasks and never succeeds on seven of the eight success-rate tasks. Between the two settings the score falls gradually as the budget widens: at η = 4, eight times the paper's width, pen is at 0.687 and relocate at 0.128. The Gaussian parameterization fails by itself, so the improvement over DSRL-SAC comes from the constraint.

AuthorsOpenReview ↗

**Response to Reviewer 1tnr (2/3)**

W3, Q2: latent mean shift and action residual

We agree that on the Adroit policies, where the decoder is almost affine, the learned shift acts on the action as a residual does. The two forms still differ in two respects.

  • Global against local. RFS [1] studies the two forms on a frozen flow-matching policy for dexterous manipulation, as a modulation of the policy's input (the latent) and of its output (an action residual). Its account is that the two do different work. A shift of the latent changes the behaviour globally and can move the policy between the modes it learned from the demonstrations. An action residual makes fine-grained corrections around the action the policy already produces and does not induce such a change.
  • Control authority. The decoder is not affine in general. On the robomimic policies it is far from affine (lift's is low-rank and bimodal in the gripper, Sec. 4.2), and there a latent shift is not a residual. Where the decoder is affine, the two forms still differ in authority. The decoder scales the latent and shifts it by an offset that depends on the state, so a latent shift acts with the gain of the pretrained policy itself, and its authority extends over the range of actions the policy generates at that state. A residual has to be held within a small bound in action units. Policy Decorator [2] reports that such a bound is essential: with a wide bound or none, the residual overrides the pretrained policy and learning degrades or fails.

Q1: the role of the constraint in TD critic learning

On pen and relocate we measured the accuracy of the critic at the final checkpoints, with and without the constraint on the same actor. At 200 visited states per checkpoint, we compare the value the critic assigns to the policy's own latents with the Monte-Carlo return of the same latents, obtained by executing them and following the policy to the end of the episode. The return is computed as the critic's own target, with the same discount and entropy term. ‖w‖²/d is the energy of the policy's latents: 1 for a prior draw and 36 with every coordinate at the latent clamp.

TaskMethod‖w‖²/dCritic valueReturn
penOurs1.105684 ± 4772447 ± 23
Same actor, no KL constraint34.5925074 ± 97460 ± 13
DSRL-SAC2.3142573 ± 261147 ± 72
Ours with a distilled critic1.103749 ± 562413 ± 10
relocateOurs1.093400 ± 2931381 ± 1
Same actor, no KL constraint35.0612779 ± 3576183 ± 1
DSRL-SAC2.114935 ± 991−220 ± 130
Ours with a distilled critic1.083620 ± 51843 ± 3
  • Scale. With the constraint the TD critic is 2.3 to 2.5 times the return, close to the distilled critic at 1.6 to 2.0 times. Without the constraint it is 70 to 420 times the return. DSRL-SAC's critic is 37 times the return on pen and positive against a negative return on relocate.
  • During training. From 25% to 100% of training, the gap between critic value and return shrinks with the constraint (pen 8404 → 3236, relocate 4259 → 2019) and grows without it (17951 → 25013, 5527 → 12596).
  • Ranking. DSRL-SAC's critic values the policy's mean latent above prior draws at more than 99% of states on both tasks, while the Monte-Carlo returns rank it above them at 35% and 22% of states. For Typical DSRL the critic does so at 85% and 70% of states and the returns at 70% and 68%.

Where the improvement comes from. A TD critic is fit only on the latents the policy has executed, and the actor moves toward the latents the critic values most. The critic's values elsewhere are extrapolations, and their errors are corrected only once the policy executes those latents. Without the constraint, the actor follows these errors out of the typical set to the latent clamp, where the frozen policy decodes actions that earn almost no return. The TD target is then evaluated at such latents too, so the overestimate grows. With the constraint, every latent the critic scores is a prior draw shifted by a bounded amount. The critic is queried next to the latents it was fit on, they decode to actions the frozen policy was trained to produce, and the returns they earn keep the critic's values in check.

AuthorsOpenReview ↗

**Response to Reviewer 1tnr (3/3)**

Q3: why a mean latent trained on typical samples performs well

The constraint and the deployment act on different latents. The constraint governs the latents the learner trains on and the critic scores, which are the samples Δθ(s) + ε. These stay typical. The deployed latent Δθ(s) is the centre of these samples.

  • Why the centre is a good latent. The samples carry the learned shift plus the prior's noise. Decoded, the noise scatters the action around the decode of Δθ(s) by more than the learned shift moves it (Fig. 6), and this scatter costs success in closed loop: success falls as the noise grows, and the mean of eight decoded samples recovers the success of the deployed latent (Fig. 7). The deployed latent keeps the shift and drops the noise.
  • How the constraint helps. The shift Δθ(s) is learned from the critic's gradient at the samples, so it is only as good as the critic is there. With the constraint the samples are typical and the critic is accurate on them (Q1). Without it, the same actor deployed in the same way is below the pretrained policy on every task (table under W2). The noise can be removed at deployment because the constraint made the shift reliable during training.

Typicality is therefore a requirement on the latents the learner trains on and the critic scores, and not on the latent that is deployed.

References

[1] Entong Su, Tyler Westenbroek, Anusha Nagabandi, and Abhishek Gupta. RFS: Reinforcement learning with residual flow steering for dexterous manipulation, ICLR 2026.

[2] Xiu Yuan, Tongzhou Mu, Stone Tao, Yunhao Fang, Mengke Zhang, and Hao Su. Policy Decorator: Model-agnostic online refinement for large policy model, ICLR 2025.

Thank you for the detailed responses and additional experiments. I particularly appreciate the controlled ablation using the same Gaussian actor with and without the KL constraint, as well as the analysis of TD critic overestimation. These results substantially address my previous concerns and provide more convincing evidence for the importance of constraining the latent policy during training.

I also find the new comparisons of different KL budgets helpful, and the explanation of deterministic deployment is now clearer to me.

I still have some reservations about the novelty and significance of the proposed method, particularly regarding what Gaussian typicality offers beyond conventional trust-region regularization and the unique benefits of latent mean shifts over action-space residuals. However, I appreciate the simplicity of the method, the substantial computational savings, and the additional insights into critic stability.

Overall, the new experiments and clarifications have improved my assessment of the paper, and I will raise my rating to 3.

Reviewer_56EtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to steer the initial noise of a frozen diffusion or flow-matching policy with RL (following the DSRL framework). Using a noise probe, the authors first show that frozen policies carry noise atypicality (norm inflation, box corners, temporal correlation) into actions, and the tolerance shrinks as the latent dimension dd grows. Based on this, they parameterize the latent policy as N(Δθ(s),Id)\mathcal{N}(\Delta_\theta(s), I_d), so its KL divergence to the prior is exactly 12∥Δθ(s)∥2\frac{1}{2}\left\lVert \Delta_\theta(s) \right\rVert^2, and constrain its average KL by η2d\eta\sqrt{2d} via a Lagrange multiplier in SAC. The authors argue that by keeping sampled latents close to the prior, a standard TD critic suffices without distilling an action critic through the decoder (as in DSRL-NA and LP-DS). At deployment, the policy acts with the mean Δθ(s)\Delta_\theta(s) instead of a sample. Evaluated on 11 simulated tasks from robomimic, Adroit, and Gym with a fixed η=0.5\eta=0.5, the method achieves a good trade-off between performance and training wall-clock time.

Critical strengths
  1. The noise probe in Sec. 3.1 is a useful finding in its own right. It shows, in both open and closed loop, that frozen diffusion policies pass atypical noise into their actions, with evidence suggesting greater sensitivity at higher latent dimensions (e.g., 1.5w retains 81% of the original return at d=12 but achieves zero success in the reported evaluation at d=112). This provides a concrete motivation for why raw-unit noise bounds may not transfer well across dimensions, which is relevant to latent steering beyond the proposed method itself.

  2. The method is simple and computationally efficient. It is essentially DSRL-SAC with a fixed-variance mean-shift policy and an exact KL constraint, retaining similar per-update computational cost while avoiding the additional decoder evaluations required for critic distillation. Despite its simplicity, it achieves competitive performance against methods that require 2 to 24 times more training time, making it a potentially useful practical baseline for this line of work.

Critical weaknesses
  1. The headline performance comparison mixes different deployment regimes. Typical DSRL is deployed with the mean latent Δθ(s)\Delta_\theta(s), while LP-DS, LP-DS-SAC, and DICE-RL are evaluated using their native stochastic policies. Although comparing methods as deployed is reasonable, Fig. 5 shows that deterministic deployment alone provides substantial gains. Under stochastic evaluation (Table 9), Typical DSRL falls clearly behind on door, hammer, and relocate. In particular, LP-DS-SAC achieves higher success rates on all three tasks while requiring slightly less training time (Tables 9–10). Thus, the claimed Pareto optimality across all 11 tasks does not hold under stochastic evaluation. This matters because the benefits of deterministic deployment are not clearly separated from those of the proposed typicality constraint.

  2. The improvements are not sufficiently attributed to the proposed mechanism. Compared with DSRL-SAC, the method simultaneously changes the policy parameterization, introduces the KL constraint, fixes the temperature, and changes deployment. The "no constraint" comparison in Fig. 4 uses DSRL-SAC rather than the same mean-shift actor without the constraint. The proposed 2d\sqrt{2d} budget scaling is also not directly compared against alternative scaling choices under the same actor. Furthermore, there is no comparison between TD and distilled critics using the same Typical DSRL actor, so the claim that typical latents make a TD critic sufficient is not directly established. With at most two seeds per cell, the evidence is insufficient to determine how much of the improvement comes from typicality itself. This is consequential because typicality is the central claimed contribution of the paper.

Questions for authors
  1. How do the baselines compare under lower-variance deployment strategies, e.g., averaging several decoded actions for LP-DS and DICE-RL? Given the substantial gains from deterministic deployment (Fig. 5), this would help assess the robustness of the headline performance comparison.

  2. Can the authors ablate the typicality constraint while keeping the mean-shift actor and other settings unchanged, including removing the KL constraint entirely and replacing the 2d\sqrt{2d} budget scaling with one proportional to dd? This would help isolate the contribution of the proposed constraint.

  3. How does the Typical DSRL actor perform with a distilled critic instead of a TD critic? This would more directly test the claim that typicality makes TD critics sufficient.

  4. Appendix B.3 states that LP-DS uses c=0.05c=0.05 on can, square, and Gym, taken from the release's lift settings, whereas the original LP-DS paper reports values of 0.35–0.5 for these tasks. Do the conclusions hold with the paper's values?

3 comments
AuthorsOpenReview ↗

**Response to Reviewer 56Et (1/2)**

We thank the reviewer for the constructive feedback and recognition of our noise probe as a useful finding in its own right and of the simplicity and computational efficiency of our method. We carefully address each concern below with additional evaluations and ablations.

W1, Q1: deployment regimes

We agree with the observation. Under stochastic evaluation, LP-DS-SAC achieves higher success rates than Typical DSRL on door, hammer and relocate. The Pareto statement therefore holds for the methods as deployed, not when every method is sampled with Gaussian noise.

For a like-for-like comparison we deployed the baselines at zero input noise, the counterpart of our maximum-likelihood latent action. LP-DS-SAC applies its correction to the zero latent, and DICE-RL adds its residual to the action decoded from the zero latent. DSRL-SAC is already deployed deterministically in the paper. LP-DS and DSRL-NA, which train a distilled critic and take longer to train while showing good performance are omitted. The table gives the results under three deployment rules: native deployment of the method (zero noise for Typical DSRL, sampled noise for LP-DS-SAC, the best of 16 samples under its critic for DICE-RL), zero noise for every method, and the mean of 8 action chunks decoded from 8 noise samples at each step. Every cell evaluates the final checkpoints on 200 episodes (transport 100) and the same initial states for every method.

DeploymentTaskOursLP-DS-SACDICE-RL
As deployedlift0.9050.7300.942
transport0.6900.3800.340
door0.9980.8730.992
hammer0.9650.8180.998
pen0.9920.8950.952
relocate0.9920.8201.000
Zero noiselift0.9050.3670.095
transport0.6900.0150.040
door0.9980.8051.000
hammer0.9650.9951.000
pen0.9920.7600.917
relocate0.9920.6601.000
Mean of 8lift0.9070.9350.833
transport0.6900.4350.400
door0.9820.9821.000
hammer0.9780.9981.000
pen0.9820.9780.867
relocate0.9850.9750.992
  • At zero noise. Typical DSRL has the highest score on lift, transport and pen, is within 0.01 of the highest on door and relocate, and lies on the Pareto frontier of score against training time on five of the six tasks. The exception is hammer, where LP-DS-SAC and DICE-RL are above ours.
  • Removing the noise is not a gain by itself. Relative to its own deployment rule, LP-DS-SAC falls on five of these six tasks and DICE-RL on three. The same actor as ours trained without the KL constraint scores 0.000 with its maximum-likelihood latent action on 7 of 8 success-rate tasks (next point). So the gain of deterministic deployment depends on the actor as Typical DSRL learns the best latent action at the given state whereas LP-DS learns a shift from the sampled noise.
  • Averaging decoded actions. We also averaged 8 decoded action chunks per step, as the reviewer suggests. Averaging raises LP-DS-SAC on all six tasks and brings it within 0.02 of Typical DSRL on door and relocate and above it on hammer. The cost is paid at deployment. Every control step runs the full generation process of the pretrained diffusion or flow policy eight times, where Typical DSRL runs it once. Even with eight generations per step, LP-DS-SAC remains below Typical DSRL's single maximum-likelihood action on four of the six tasks (transport, door, pen and relocate) and is above it on lift and hammer. Typical DSRL gains nothing from the same averaging (its scores move by at most 0.02), since its maximum-likelihood latent action already removes the sampling noise in one generation. LP-DS-SAC has no single-generation rule with this effect: zero noise lowers it on five of the six tasks.
AuthorsOpenReview ↗

**Response to Reviewer 56Et (2/2)**

W2, Q2: attribution to the constraint

Same actor without the constraint. We trained Typical DSRL with only the KL constraint removed: the same mean-shift actor with σ = 1, the same fixed temperature, the same TD critic and the same maximum-likelihood deployment. It scores 0.000 on lift, can, square, transport, door, hammer and relocate, 0.092 on pen, and 183 / −8 / 826 on hopper / walker / halfcheetah. This is below the frozen pretrained policy on every task. Its latents sit at the clamp: ‖w‖²/d is 34–35 on pen and relocate, against 1 for a draw from the prior.

Widening the budget. On the same actor we swept the budget width η:

ηhopper (d = 12)pen (d = 192)relocate (d = 240)
0.12531700.9370.990
0.2531480.9680.995
0.5 (paper)31710.9880.997
131660.9750.947
218850.8570.612
411180.6870.128
no constraint1830.0920.000

Performance falls steadily as the budget widens beyond η = 1 and ends at the unconstrained run.

√(2d) against a budget proportional to d. The two rules are set equal at d = 28 (lift, can, square), so they differ only away from it:

Taskd√(2d) budget (ours)budget ∝ d
hopper123171 ± 63167 ± 8
transport1120.647 ± 0.0520.365 ± 0.160
pen1920.988 ± 0.0020.962 ± 0.005
relocate2400.997 ± 0.0000.693 ± 0.003

The rules tie at d = 12, and the proportional budget is too wide at d ≥ 112.

Seeds. We ran Typical DSRL on seeds 1–5 on all eleven tasks (1–4 on transport) with one setting for every task (η = 0.5, α = 0.05), and LP-DS-SAC on seeds 1–5 on Adroit and Gym. The five-seed means of Typical DSRL are within 0.01 of the two-seed means on every success-rate task except transport (0.695 vs 0.647), and within 0.5% on Gym.

TaskOursLP-DS-SAC
door0.989 ± 0.0060.932 ± 0.019
hammer0.976 ± 0.0060.850 ± 0.008
pen0.979 ± 0.0120.889 ± 0.018
relocate0.993 ± 0.0040.834 ± 0.028
hopper3163 ± 172847 ± 13
walker3986 ± 443698 ± 32
halfcheetah4972 ± 64557 ± 26

Over five seeds, Typical DSRL is higher than LP-DS-SAC on all seven tasks by more than the seed spread.

Q3: Typical DSRL with a distilled critic

We trained the Typical DSRL actor and constraint with DSRL-NA's distilled critic in place of the TD critic. Training hours are for both critics on the same machine.

TaskDSRL's box constraint: TD → distilled (DSRL-SAC → DSRL-NA)our constraint: TD → distilledtraining hours, TD / distilled
lift0.897 → 0.9550.883 → 0.9755.0 / 12.7
square0.460 → 0.9220.920 → 0.95847.7 / 61.7
pen0.582 → 0.9270.988 → 0.99215.0 / 39.4
relocate0.030 → 0.9450.997 → 1.00018.1 / 38.2

The distilled critic is at or above the TD critic on all four tasks. On square, pen and relocate it adds at most 0.04 under our constraint, where it adds 0.35–0.92 under DSRL's box, and it costs 1.3–2.6 times the training time. On lift the gap is 0.09. The constraint therefore removes most of the benefit of distillation, not all of it. Consistent with this, the TD critic's overestimation at the policy's own latents (critic value minus Monte-Carlo return) shrinks during training with the constraint (pen 8404 → 3236, relocate 4259 → 2019) and grows without it (17951 → 25013, 5527 → 12596).

Q4: the LP-DS paper's c

We re-ran LP-DS-SAC at the LP-DS paper's c on four of the six tasks where our runs used 0.05 (the LP-DS paper also lists 0.10 for lift). We use LP-DS-SAC because it is the variant closest to Typical DSRL in training time; full LP-DS trains 2.6–7 times longer than Typical DSRL on these tasks (Table 10).

Taskpaper's cLP-DS-SAC, c = 0.05LP-DS-SAC, paper's cOurs
lift0.100.798 ± 0.0080.807 ± 0.0680.883 ± 0.067
hopper0.52849 ± 62486 ± 683175 ± 25
walker0.353683 ± 33943 ± 1373925 ± 17
halfcheetah0.354528 ± 15015 ± 195015 ± 2

The paper's c leaves LP-DS-SAC unchanged within its seed spread on lift and lowers it on hopper, so the conclusions hold there. It raises LP-DS-SAC on walker and halfcheetah, where LP-DS-SAC now matches Typical DSRL.

Thank you for the thorough rebuttal. The same-actor ablation, η\eta sweep, and five-seed results address much of my original attribution concern, and I have raised my score.

For the final version, I would encourage the authors to: (1) Rerun LP-DS-SAC (and LP-DS where feasible) on can and square with the original LP-DS paper's cc values, and revisit Figure 2 and the Pareto claim, especially given that LP-DS-SAC matches Typical DSRL on walker2d and halfcheetah with the paper's settings. (2) Emphasize the mean-of-8 comparison as a more meaningful lower-variance deployment baseline, while accounting for its additional inference cost. (3) Qualify the TD-critic claim, given the remaining benefit of distillation on lift and square.

I appreciate the simplicity and efficiency of the proposed method, and the rebuttal has genuinely strengthened my confidence in its empirical results. My remaining hesitation is mainly about significance: I still see this as a solid but incremental advance within latent steering. Nevertheless, I recognize its practical value.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers & Summary of Revisions

We sincerely thank all reviewers for the thoughtful feedback and the follow-up discussion, which have helped us strengthen the rigor and depth of the paper. We are encouraged that the reviewers recognize:

  • A useful diagnosis: the noise probe shows that frozen diffusion policies carry atypical noise into their actions, with a tolerance that shrinks as the latent dimension grows (56Et, 1tnr, yLpA).
  • A simple method: a mean-shift latent policy whose KL divergence to the prior is exact and cheap to compute, with no update of the pretrained model (56Et, 1tnr, yLpA).
  • Computational efficiency: performance comparable to or better than the baselines at a fraction of their training cost, without critic distillation (56Et, 1tnr, yLpA).
  • Thorough and fair experiments: eleven tasks, every method run in the same training loop, and analyses of the mechanism (1tnr, yLpA).
  • Clear presentation: the paper is clear and easy to follow (56Et, yLpA).

Overall, this work shows that a frozen generative policy acts well only on noise typical of its prior, and turns this observation into a single constraint, an exact KL budget stated in widths of the prior's typical shell, under which a latent policy trained with a plain temporal-difference critic matches the strongest baselines at the cost of DSRL-SAC.

We have carefully responded to each reviewer's concerns with additional experiments, analyses and clarifications. The revised paper incorporates the updates summarized below, primarily in Section 4.2 for the main findings and Appendices C.5–C.10 for the additional experiments:

  • Attribution to the constraint: training the same actor with only the KL constraint removed on all eleven tasks (56Et: W2/Q2; 1tnr: W2/Q1; yLpA: W1/W2), sweeping the budget width η on hopper, pen and relocate (56Et: Q2; yLpA: W2), and comparing the √(2d) budget with a budget proportional to d and a KL penalty of fixed weight (56Et: Q2; 1tnr: W1; yLpA: W2).

    Revisions: Section 4.2; Appendix C.5, Tables 13–15.

  • Deployment-controlled comparison: evaluating the baselines at zero input noise and with the mean of 8 decoded action chunks, and stating the inference cost of averaging, eight generations of the pretrained policy per control step against one (56Et: W1/Q1; yLpA: W1/Q1).

    Revisions: Section 4.2; Appendix C.6, Table 16.

  • Temporal-difference critic: training Typical DSRL with a distilled critic (56Et: Q3) and measuring the accuracy of its critic against Monte-Carlo returns with and without the constraint (1tnr: Q1). Given the remaining benefit of distillation on lift and square, the paper now states that typical latents make a temporal-difference critic largely sufficient (56Et).

    Revisions: Abstract; Sections 1, 4.2 and 5; Appendix C.7, Tables 17–18.

  • Baseline settings and robustness: rerunning LP-DS-SAC at the LP-DS paper's c on all six affected tasks, and LP-DS where it completed (56Et: Q4), training Typical DSRL at one temperature on every task (yLpA: Q2), and extending Typical DSRL and LP-DS-SAC to five seeds (56Et: W2; yLpA).

    Revisions: Section 4.1; Appendix B.3; Appendix C.8, Table 19; Appendix C.9, Table 20; Appendix C.10, Table 21.

  • Relation to prior work: discussing Noise PPO, FIND and RFS (1tnr: W1, W3/Q2).

    Revisions: Section 2.

We genuinely appreciate the reviewers' engagement throughout the discussion, and hope that the improvements made during the rebuttal period support their final evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.