acceptodds
Under review as a conference paper at ICLR 2027

DePO: Mitigating Policy Entropy Collapse via Denoised Policy Optimization

Abstract

Reinforcement learning has become a central tool for improving large language model reasoning, but rapid policy entropy decay can restrict exploration during training. We propose Denoised Policy Optimization (DePO), which filters importance sampling ratios along token positions and uses the filtered values as detached weights in a policy update with sign-dependent clipping gates. We study sliding-window averaging, exponentially weighted averaging, and Savitzky–Golay filtering as simple ways to control local ratio fluctuations. Building on prior analyses of entropy dynamics, we derive the detached-weight gradient and characterize noise reduction and smoothing bias under explicit assumptions. Experiments on mathematical reasoning and code generation cover three model backbones. On Qwen3-4B, DePO improves the reported average math accuracy from 65.5 to 68.4 and code accuracy from 39.0 to 42.3, while retaining higher final policy entropy than GRPO. Window studies, generation statistics, and a multi-turn search evaluation complement filter ablations and clipping-relaxation stress tests. These results support ratio filtering as a practical stabilization method; its effects on individual reasoning decisions remain to be established.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, Reinforcement Learning, Deep Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QDtyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

DePO modifies GRPO by smoothing the per-token importance ratios within each response with a simple filter (moving average, exponential average or Savitzky–Golay), and uses the smoothed ratios as fixed weights in the update. The aim is to reduce noisy updates and slow entropy collapse. Experiments on math and code with Qwen3-1.7B, Qwen3-4B and LLaMA-3.2-3B show the best average scores and higher entropy than GRPO.

Critical strengths
  • Simple, cheap method that is easy to add to GRPO.
  • The authors are honest about what their analysis does not prove.
Critical weaknesses
  • Key settings of the method are missing. The paper itself says several are "not specified in the available records," including whether the default window size of 5 is a radius or a full width (Sec. 4.1, App. B), the EWMA initialization, the seeds and the decoding settings. No code is provided.
  • The tables are described as "existing records," not a rerun; there are no seeds, confidence intervals or significance tests.
  • Gains over the best baseline are small (e.g., +0.7 on Qwen3-4B math) and may be noise.
  • Some results look implausible: LLaMA-3.2-3B-Instruct scores 56–59% on AIME24 with every method.
  • Baseline results are unusually regular, with the same ranking and nearly constant gaps on every benchmark.
  • Limited novelty: smoothing the ratios with a standard filter is a small change to GRPO.
  • Hard to read; most claims are followed by long caveats.
Questions for authors

Q1 Can you release code and specify the missing settings?

Q2 How many seeds were used in Table 1, and what is the variance?

Q3 How do you explain the LLaMA-3.2-3B AIME24 results compared with published numbers?

Q4 Were all baselines run by you with the same budget and tuning?

Reviewer_w6qkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DePO aims to reduce policy entropy collapse by smoothing importance ratios across tokens in a response. The smoothed ratios weight the policy update, with gradients stopped through the weights. Clipping depends on the advantage sign. The paper studies three filters and analyzes their noise reduction and bias. Experiments cover math and code tasks with three backbones, plus clipping, window-size, and multi-turn search studies.

Critical strengths
  1. DePO improves task scores and tolerates relaxed clipping. On Qwen3-4B, math and code averages improve over GRPO by 2.9 and 3.3 points. Without clipping, AIME25 accuracy remains at 50.2%, close to the default 51.2%, while the tested GRPO run diverges.

  2. The method is simple and the theory is carefully scoped. It requires no extra learned module or reward changes. The analysis states when filtering reduces ratio noise and explains why this does not guarantee lower gradient variance or better policy updates.

Critical weaknesses
  1. The strongest-baseline gains are hard to assess. Table 8 omits code training steps and the KL coefficient, preventing a check of matched training conditions. Independent training seeds are also missing. This matters because the math gains over SIREN are only 0.7 points on Qwen3-4B and 0.5 on Llama-3.2-3B.

  2. The benefit of local filtering needs a stronger comparison. Tables 5 and 6 compare filters and window sizes, but leave open whether local neighborhoods matter. GSPO is discussed but not evaluated. Its sequence-level importance ratios and clipping make it a relevant comparison, although it is not a token-ratio filtering method. A sequence-wide aggregation control within DePO would more directly test whether local windows add value.

Questions for authors
  1. Is the default SWA window of 5 a radius or full width? Please provide the filtering and training configurations, including boundary/mask handling, updates per rollout batch, and the missing code settings. Were training conditions matched with SIREN?

  2. Could you report independent training-seed variability for one representative DePO/SIREN comparison?

  3. Could you compare DePO with GSPO using matched data and training budgets, with appropriate hyperparameters for each method? This would assess performance against a related sequence-level approach. Separately, a control that changes only DePO's ratio aggregation from local windows to the whole response would isolate the role of locality.

Reviewer_47LVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper argues that rapid entropy decay can restrict exploration. In order to fix that issue, this paper presents DePO, which filters important sampling ratios along token positions and uses the values as weights in policy updates. For each response, Depo computes the probability ratio of each token, filters the sequence using a sliding-window average, exponentially weighted average or Savitzky-Golay filter and uses the ratio as weights. The main reason behind doing this is to smooth out ratio fluctuations provided by neighboring tokens to stabilize training. Experiments are done using Qwen3 and Llama 3.2 families with various sizes.

Critical strengths

Paper does present experiments across multiple tasks, showing that the method can be generalized.

Critical weaknesses

The comparison doesn't seem to be valid, since it is better to compare against existing important-weight stabilization method, such as KPO, CISPO, GMPO and GSPO I also am a bit curious why it makes sense to denoise the fluctuations. Maybe those fluctuations are beneficial in certain conditions and data sets. There should be better ablation study to show this.

Misc. There are few missing details around the algorithm and parameter details, so it would be hard to replicate this work. Confidence intervals are not provided, so not sure if this is just one run that is better.

Questions for authors

Have you seen similar works like He et al., Online Causal Kalman Filtering for Stable and Effective Policy Optimization? The idea seems very similar. Also MiniMax CISPO uses similar technique.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time and thoughtful reviews. Please read the authors’ response and share your follow-up comments promptly, indicating whether your concerns have been addressed. Please raise any additional questions early enough to give the authors sufficient time to respond.

Thank you.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.