- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper tackles the problem of importance ratio control in RLVR. The authors point out that hard clipping in GRPO completely removes gradients in the tail, while removing clipping causes instability from extreme ratio weights. They further argue that in autoregressive generation, repeated ratio contractions () accumulate multiplicatively along the prefix and suppress valid reasoning paths. To address this, they propose MHPO, which replaces standard clipping with two operators: the Log-Fidelity Modulator (LFM) and the Decoupled Hazard Penalty (DHP).
LFM maps the log importance ratio through a scaled tanh, , so that the gradient multiplier stays smooth, positive, and globally bounded. DHP then applies a detached survival weight derived from Weibull cumulative hazard functions with softplus inputs. This weight attenuates ratio expansion and contraction asymmetrically, with stronger decay on the contraction side.
Critical strengths
Good training stability. While the peak performance gain is somewhat marginal, MHPO prevents the late-stage performance collapse, only a 0.8% drop from the best to the latest checkpoint, compared to 6.9-12.2% in the baselines.
Critical weaknesses
-
Theorem 1 does mean NOTHING. This is obvious results by tanh.
-
LFM is contradicting to DHP, i think. The Log-Fidelity Modulator is introduced to retain the tail signals that clipping discards, while the DHP suppresses those tail signals again. Why is the multi-step composition of logarithm, tanh, softplus, and the Weibull power required rather than a pre-defined S-curve, at the cost of five hyperparameters in place of a single ε?
Compared with CISPO, which has a similar philosophy, I cannot find a significant benefit of the suggested algorithm considering its complexity, especially in Table 2.
Refs
[1] Fedus, William, et al. "Hyperbolic discounting and learning over multiple horizons." arXiv preprint arXiv:1902.06865 (2019).
Questions for authors
How is hazard analysis actually related to the suggested structure?
As shown in Fedus et al. [1], discounting itself is related to the survival function of when an RL episode ends. The change in the token probability ratio has nothing to do with this uncertainty, as this paper simply focuses on a bell-shaped function. I think this is just smoothing that contracts rather than survival function analysis; if it were truly related to survival analysis, it should be a function of the token position (episode time).
Page 5, lines 245-253 are not convincing, as it is just introduced as a shape of a function rather than for some probabilistic reason.