- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a new algorithm TAPO. The paper first identifies the state distribution approximation in TRPO/PPO/GRPO works, and demonstrates that such approximation is not negligible in LLM post-training. The paper then derives prefix-level importance sampling that correctly fixes the state distribution mismatch in the objective, apply standard empirical transformations like geometric average to reduce computation instability, and demonstrate the advantage of TAPO over the other existing methods.
Critical strengths
-
The motivation experiments in figure 1 clearly demonstrates the state distribution shift problem. Previous algorithms (PPO, GRPO family) have been taking the state distribution approximation in PPO as granted, while such shift problem has been rarely visualized.
-
The design of the algorithm is well-motivated and easy to understand. Each design component of the algorithm is either principled (e.g., using state distribution importance sampling) or a widely-accepte practice (e.g., using geometric average, length adaptive clipping).
-
In experiments, a large number of existing algorithms are tested as baselines to support the findings. A wide variety of ablation studies were conducted.
Critical weaknesses
-
The pharsing "achieving up to 48% and 12% improvement" is misleading. Most papers state absolution accuracy improvement instead of relative improvement in percentage. Such phrasing is eyecatching but causes misinterpretation.
-
In evaluation, Avg@8 still bears noise in accuracy. Having more responses per problem during evaluation is preferred, e.g. Avg@32.
-
The novelty of the paper is moderate but acceptable. Prior works such as [1] have identified the same problem in LLM post-training.
[1] A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization
Questions for authors
Do authors have comments on the weaknesses listed above?