acceptodds
Withdrawn from ICLR 2027

MHPO: Modulated Hazard-aware Policy Optimization for Stable Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Controlling importance ratios in reinforcement learning with verifiable rewards requires retaining useful learning signals while limiting the influence of extreme policy deviations. Hard clipping can eliminate direct gradients for informative tokens, while unbounded ratio weighting can overemphasize outliers. We propose Modulated Hazard-aware Policy Optimization (MHPO), which separates these concerns through its gradient multipliers. A Log-Fidelity Modulator (LFM) preserves local ratio fidelity and supplies a smooth, positive, globally bounded multiplier. A Decoupled Hazard Penalty (DHP) uses established cumulative hazard functions to attenuate ratio expansions and contractions separately, without reversing their direct gradient contributions. Under an explicit assumption on the joint moment of advantages and score norms, we derive a local bound on the second moment of the gradient. We also explain how repeated ratio contractions accumulate along autoregressive prefixes. Component ablations, token masking, and diagnostics of directional tails connect these design choices to retained learning signals and reduced prefix contraction. Among the evaluated baselines, MHPO achieves the best average across benchmarks on both Qwen3-4B-Base and Qwen3-30B-A3B-Base. Further evaluations on instruction-tuned and multimodal models support transfer across the tested settings

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_T5EEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper tackles the problem of importance ratio control in RLVR. The authors point out that hard clipping in GRPO completely removes gradients in the tail, while removing clipping causes instability from extreme ratio weights. They further argue that in autoregressive generation, repeated ratio contractions (rt<1r_t < 1) accumulate multiplicatively along the prefix and suppress valid reasoning paths. To address this, they propose MHPO, which replaces standard clipping with two operators: the Log-Fidelity Modulator (LFM) and the Decoupled Hazard Penalty (DHP).

LFM maps the log importance ratio through a scaled tanh, c⋅tanh(log⁡rc)c·\text{tanh}(\log \frac{r}{c}), so that the gradient multiplier stays smooth, positive, and globally bounded. DHP then applies a detached survival weight derived from Weibull cumulative hazard functions with softplus inputs. This weight attenuates ratio expansion and contraction asymmetrically, with stronger decay on the contraction side.

Critical strengths

Good training stability. While the peak performance gain is somewhat marginal, MHPO prevents the late-stage performance collapse, only a 0.8% drop from the best to the latest checkpoint, compared to 6.9-12.2% in the baselines.

Critical weaknesses
  1. Theorem 1 does mean NOTHING. This is obvious results by tanh.

  2. LFM is contradicting to DHP, i think. The Log-Fidelity Modulator is introduced to retain the tail signals that clipping discards, while the DHP suppresses those tail signals again. Why is the multi-step composition of logarithm, tanh, softplus, and the Weibull power required rather than a pre-defined S-curve, at the cost of five hyperparameters in place of a single ε?

Compared with CISPO, which has a similar philosophy, I cannot find a significant benefit of the suggested algorithm considering its complexity, especially in Table 2.

Refs

[1] Fedus, William, et al. "Hyperbolic discounting and learning over multiple horizons." arXiv preprint arXiv:1902.06865 (2019).

Questions for authors

How is hazard analysis actually related to the suggested structure?

As shown in Fedus et al. [1], discounting itself is related to the survival function of when an RL episode ends. The change in the token probability ratio rtr_t has nothing to do with this uncertainty, as this paper simply focuses on a bell-shaped function. I think this is just smoothing that contracts log⁡rj\log r_j rather than survival function analysis; if it were truly related to survival analysis, it should be a function of the token position (episode time).

Page 5, lines 245-253 are not convincing, as it is just introduced as a shape of a function rather than for some probabilistic reason.

Reviewer_ELgwOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Modulated Hazard-aware Policy Optimization (MHPO) for reinforcement learning with verifiable rewards. Its Log-Fidelity Modulator (LFM) smoothly bounds importance-ratio gradient multipliers while retaining learning signals beyond clipping boundaries. A detached Decoupled Hazard Penalty (DHP) separately dampens ratio expansions and contractions. The authors derive a conditional bound on the gradient's second moment and report improved mathematical reasoning accuracy and better retention of performance through the final checkpoint across several backbones.

Critical strengths
  1. The method has a clear design, with separate components for smooth ratio control and directional damping. Section 3 distinguishes ratio direction from advantage sign and explains why detaching the hazard penalty preserves the direction of each token's direct gradient contribution. The multiplier bound makes the control of ratio-induced amplification explicit.

  2. The experiments help explain why the method works. Section 4.3 tests targeted token masking against matched shuffled masks and compares DHP with uniform attenuation matched to its update norm. These controls support the value of retaining clipped-tail signals and suggest that directional damping helps beyond simply shrinking updates. MHPO also outperforms CISPO on Qwen3-4B and loses substantially less performance between its best and final checkpoints.

Critical weaknesses
  1. The theory provides limited justification for asymmetric hazard attenuation. Theorem 1 bounds the gradient's second moment under an assumption on the joint moment of advantages and score norms, but the same bound holds for LFM alone and does not depend on the DHP parameters. As Appendix C.2 acknowledges, contraction-side damping also weakens corrective updates for positive-advantage tokens whose ratios are below one.

  2. The paper does not report variability across training seeds or confidence intervals. This leaves uncertainty about the limited gains over CISPO.

Questions for authors
  1. Could the authors add ablations that separate tokens by advantage sign and ratio direction? This would help clarify when stronger contraction damping is beneficial and when it slows corrective updates for positive-advantage tokens whose ratios are below one.
Reviewer_G3sHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Modulated Hazard-aware Policy Optimization (MHPO), a modification of GRPO for reinforcement-learning fine-tuning of language models. It combines smooth importance-ratio modulation (LFM) with asymmetric gradient attenuation (DHP) to retain useful learning signals while limiting extreme policy updates. The method is evaluated primarily on mathematical reasoning benchmarks across several model configurations.

Critical strengths
  • MHPO improves average accuracy over CISPO from 48.5% to 52.9% on Qwen3-4B and from 54.0% to 54.7% on Qwen3-30B-A3B. The smaller-model improvement is meaningful.

  • The paper evaluates LFM and DHP separately, examines hyperparameter sensitivity, and compares symmetric and asymmetric attenuation. It also reports that removing gradient contributions from tokens that would be clipped under GRPO reduces performance more than removing a matched, shuffled selection of contributions, supporting the usefulness of the retained learning signals.

Critical weaknesses
  • The paper does not clearly report independent training seeds, repeated runs, or uncertainty estimates. This limits confidence in the reported accuracy improvements and stability results.

  • MHPO's advantage over CISPO falls from 4.4 percentage points on Qwen3-4B to 0.7 points on Qwen3-30B-A3B. The experiments do not establish whether the difference is related to model size, architecture, or hyperparameter choices.

  • Gradient norms alone are insufficient to demonstrate stable optimization, especially when the proposed method directly attenuates gradients. Checkpoint-retention results are more informative, but are not supported by reported variability across training runs.

  • The main experiments focus on mathematical reasoning and Qwen-derived models. Generalization to other tasks and model families remains uncertain.

Questions for authors
  • How many independent training runs were conducted? Can the authors report seed-level variability and uncertainty intervals for benchmark accuracy and training stability?

  • Why does MHPO's advantage over CISPO become much smaller on Qwen3-30B-A3B? Does this reflect model scale, architecture, or hyperparameter choices?

  • Can the method's benefits be demonstrated on non-mathematical reinforcement-learning tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.