- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper splits the error of an off-policy RL update for LLMs into two parts: bias from scoring the new policy on prefixes written by the old one, and variance from estimating it on a finite batch. They argue that the χ² divergence between the new and old policy controls both, while KL does not. Based on this they propose CHIP, which replaces PPO clipping with a projection: whenever a next-token distribution goes past a χ² limit from the old policy, it is moved to the closest distribution within that limit, and a KL term pulls the actor toward this projected distribution. The authors claim the experiment results on math RL show CHIP method outperforms clipping.
Critical strengths
-
Writing and framing are good. Splitting the update error into these two layers and showing that one quantity covers both is clean, and the paper explains it well. The example with a 1e-6 probability token makes it easy to see why a KL budget can miss what χ² catches.
-
The method is interesting. It only replaces the clipping step without affecting the advantages of the classic RL algorithm, and the cost is low.
Critical weaknesses
-
The closest prior work is TROLL (Becker et al., 2026), which already replaces clipping with a differentiable trust region projection on a sparse set of tokens and trains the actor toward the projected distribution. As far as I can see CHIP keeps the same recipe and swaps KL for χ². But TROLL is only cited in passing in Sec 3.4 and is not compared against.
-
Most of the gains come from runs where GRPO peaks and then falls apart (at 4B, it loses about half its accuracy). At 32B, where GRPO trains fine, the gain is around one point. Every setting is run once, and ε is also changed for the larger models. It can’t tell if CHIP performs better.
Questions for authors
-
Can you compare against TROLL? It already does the same kind of projection with KL, so this is the most direct way to show that χ² is actually better than KL.
-
Can you compare against a GRPO run that doesn’t collapse? Right now most of the gain seems to come from GRPO falling apart, so I’d like to see if CHIP still helps when GRPO trains normally.
5 comments
Response to Reviewer PMw3 (2/3): Weakness 2 and Q2 - Stable GRPO controls
Weakness 2 and Question 2: Does CHIP's advantage depend on collapse of the GRPO baseline?
Thank you for highlighting the importance of baseline stability. CHIP's performance advantage does not require late-training collapse of GRPO. In the original 4B and 14B mathematical-reasoning settings, we continued to observe late-training instability after changing the training seed. To compare against GRPO that trains stably, we increased its reference-policy KL penalty coefficient from 0.001 to 0.003 and conducted additional experiments at these two model scales.
With the stronger KL penalty, neither GRPO run exhibits the late-training collapse seen previously, yet CHIP outperforms GRPO at all 30 common evaluation points in each comparison. We report every evaluation from steps 10 to 300 below. The metric is DAPO validation-suite mean accuracy.
Qwen3-4B mathematical reasoning
| Step | GRPO (%) | CHIP (%) | CHIP gain (pp) |
|---|---|---|---|
| 10 | 43.9 | 49.0 | +5.1 |
| 20 | 48.6 | 51.6 | +3.0 |
| 30 | 49.2 | 55.4 | +6.2 |
| 40 | 49.4 | 55.2 | +5.8 |
| 50 | 52.5 | 54.2 | +1.7 |
| 60 | 52.2 | 53.2 | +1.0 |
| 70 | 52.2 | 54.8 | +2.6 |
| 80 | 50.7 | 54.9 | +4.2 |
| 90 | 51.8 | 55.2 | +3.4 |
| 100 | 52.2 | 56.5 | +4.3 |
| 110 | 52.1 | 56.3 | +4.2 |
| 120 | 52.2 | 55.7 | +3.5 |
| 130 | 52.7 | 55.7 | +3.0 |
| 140 | 52.5 | 56.8 | +4.3 |
| 150 | 52.5 | 56.6 | +4.1 |
| 160 | 52.8 | 54.6 | +1.8 |
| 170 | 54.1 | 54.7 | +0.6 |
| 180 | 53.9 | 56.5 | +2.6 |
| 190 | 53.9 | 54.1 | +0.2 |
| 200 | 52.6 | 55.5 | +2.9 |
| 210 | 53.4 | 56.5 | +3.1 |
| 220 | 52.8 | 56.7 | +3.9 |
| 230 | 52.5 | 56.5 | +4.0 |
| 240 | 52.4 | 56.5 | +4.1 |
| 250 | 52.7 | 55.4 | +2.7 |
| 260 | 54.4 | 55.0 | +0.6 |
| 270 | 54.2 | 57.0 | +2.8 |
| 280 | 54.1 | 54.8 | +0.7 |
| 290 | 54.1 | 56.0 | +1.9 |
| 300 | 52.7 | 57.5 | +4.8 |
Qwen3-14B mathematical reasoning
| Step | GRPO (%) | CHIP (%) | CHIP gain (pp) |
|---|---|---|---|
| 10 | 44.3 | 53.8 | +9.5 |
| 20 | 51.0 | 54.4 | +3.4 |
| 30 | 51.7 | 55.8 | +4.1 |
| 40 | 52.1 | 57.0 | +4.9 |
| 50 | 52.7 | 57.6 | +4.9 |
| 60 | 52.5 | 57.8 | +5.3 |
| 70 | 55.5 | 59.0 | +3.5 |
| 80 | 55.2 | 58.5 | +3.3 |
| 90 | 55.6 | 59.7 | +4.1 |
| 100 | 54.5 | 60.0 | +5.5 |
| 110 | 54.9 | 60.3 | +5.4 |
| 120 | 56.2 | 59.7 | +3.5 |
| 130 | 56.9 | 59.1 | +2.2 |
| 140 | 57.8 | 61.5 | +3.7 |
| 150 | 57.6 | 61.1 | +3.5 |
| 160 | 57.3 | 62.0 | +4.7 |
| 170 | 58.8 | 61.8 | +3.0 |
| 180 | 58.6 | 60.5 | +1.9 |
| 190 | 57.6 | 61.3 | +3.7 |
| 200 | 58.8 | 61.7 | +2.9 |
| 210 | 58.0 | 59.7 | +1.7 |
| 220 | 59.1 | 61.2 | +2.1 |
| 230 | 57.2 | 61.7 | +4.5 |
| 240 | 57.5 | 59.8 | +2.3 |
| 250 | 59.6 | 61.4 | +1.8 |
| 260 | 57.9 | 61.9 | +4.0 |
| 270 | 57.6 | 61.7 | +4.1 |
| 280 | 59.3 | 61.3 | +2.0 |
| 290 | 59.4 | 62.1 | +2.7 |
| 300 | 59.6 | 62.4 | +2.8 |
At step 300, CHIP achieves 57.5% and 62.4% on 4B and 14B, respectively, exceeding GRPO with the stronger KL penalty by 4.8 and 2.8 percentage points. Across the seven evaluations from steps 240 to 300, CHIP's mean advantages are approximately 2.5 and 2.8 percentage points, respectively. The gains therefore persist during training rather than appearing only at a few checkpoints after baseline collapse.
Response to Reviewer PMw3 (3/3): Weakness 2 - SWE, 32B, single runs, and radius
SWE-bench Verified
Figure 2 in Section 6.3 provides further evidence: the CLIP baseline trains without pronounced late-training collapse, while CHIP still achieves better performance. This baseline uses asynchronous CLIP based on the DeepSWE recipe. CHIP remains ahead at every evaluation from step 40 onward; the table reports the last four evaluations (pass@1).
| Step | CLIP (%) | CHIP (%) | CHIP gain (pp) |
|---|---|---|---|
| 100 | 38.8 | 45.8 | +7.0 |
| 110 | 38.8 | 41.8 | +3.0 |
| 120 | 38.8 | 45.4 | +6.6 |
| 130 | 39.0 | 45.8 | +6.8 |
At step 130, CHIP achieves 45.8%, compared with 39.0% for CLIP, a gain of 6.8 percentage points; the mean gain over the last four evaluations is 5.9 points.
These results show that CHIP's advantage is present when the baseline trains stably, not only when it collapses. This is consistent with our theoretical analysis: under the stated assumptions, a χ² constraint controls both state-distribution shift error and finite-sample estimation error. Its role is therefore not limited to preventing collapse; the analysis also provides a theoretical basis for sustained performance gains during stable training.
32B, where GRPO trains stably
At 32B, CHIP's gain on the validation-suite mean is smaller, +1.2 points over the final sixty steps, but consistent: CHIP is higher at all 30 synchronized evaluations (Figure 1). On held-out AIME'25, CHIP gains +6.2 points, and DAPO training success increases by +2.4 points (Table 1). The SWE-bench Verified experiment above also uses Qwen3-32B with a stable CLIP baseline, and there CHIP leads by 6.8 points at step 130.
Single training runs
Each comparison is a separately trained matched pair, and the direction of the result is the same across the six model scales in Table 1, the seven algorithm configurations in Table 2, and the new TROLL, auxiliary-KL, and stable-GRPO controls. Within each pair, the gain is not driven by a single checkpoint: across the six scales in Table 1, CHIP is higher at 172 of 177 synchronized evaluations. The baseline collapse is also not a property of one seed: as noted above, rerunning GRPO at 4B and 14B with a different seed reproduced the late-training instability.
Choice of radius
The 0.6B–8B runs use ε=0.05, which is below the best radius at 8B (Table 3 and the ε=0.5 ablation in our response to Reviewer uPU9), so the gains at these scales were obtained without tuning ε to its best value. For 14B and 32B, we tried several radii in preliminary runs and found ε=0.25 to work well, consistent with the 8B ablation. In Table 2, both radii also improve over the clipped baseline for GSPO (+0.7 and +0.9 points) and DAPO (+3.8 and +6.7 points), so the gains do not depend on one particular radius.
Response to Reviewer PMw3 (1/3): Q1 - Relationship to TROLL
Response to Reviewer PMw3
Revised manuscript. We have uploaded a revised PDF. The changes relevant to this review are: (1) in response to the reviewer's point that TROLL was only cited in passing, a new Related Work paragraph, Trust-region projections (Section 5), that discusses TROLL directly, including the shared training objective, the KL versus χ² projection problems, and their consequence for importance-weight variance; and (2) a new Appendix I.4 that reports the SWE-bench Verified comparison with TROLL and the CLIP + auxiliary-KL control from this response.
Weakness 1 and Question 1: Relationship to TROLL
Thank you for raising this question. The reviewer is right that CHIP keeps TROLL's training objective; we made this choice deliberately and changed only the projection problem. Eq. (8) of CHIP has the same form as Eq. (5) of TROLL: both use the projected distribution in the importance ratio and regress the raw actor toward it with a stop-gradient KL term. Keeping this part fixed isolates the one component our theory is about, the divergence that defines the projection.
1 The projection problems differ
Write πθ for the raw actor output, πˉθ for the projected distribution, r=πθ/πold for the raw ratio, and rˉ=πˉθ/πold for the projected ratio:
TROLL's geometric solution passes a large raw ratio through as a large projected ratio, while CHIP's harmonic solution caps every projected ratio and bounds their variance by the budget.
2 Why the divergence matters
Proposition 2 (KL blind spot). For every ε>0 and every M>0, there exist policies π and πold such that KL(π∥πold)≤ε while Varπold(π/πold)=χ2(π∥πold)≥M.
A KL ball therefore contains updates with arbitrarily large importance-weight variance, and this variance drives the finite-batch estimation error (Layer 2). Under the stated assumptions, a χ² budget bounds this variance and also bounds the total-variation distance that controls the state-distribution-shift error (Layer 1). The χ² budget controls both error layers, whereas a KL budget controls only Layer 1.
3 Experimental comparison with TROLL
For both Qwen3-14B mathematical reasoning and Qwen3-32B on SWE-bench Verified, CHIP uses a χ² radius of ε=0.25 and an auxiliary KL coefficient of α=1. TROLL uses the published defaults in Table 3: a KL radius of ε=0.05 and a KL regression coefficient of α=1. We compared results at common evaluation steps.
The table reports the last four evaluations of each task:
Over the full runs, CHIP leads TROLL at all 30 evaluations on mathematical reasoning (steps 10–300) and at 10 of the 14 evaluations on SWE-bench Verified (steps 0–130), remaining ahead from step 40 onward. Over the last four evaluations, the mean gain is 1.9 points on mathematical reasoning and 4.8 points on SWE-bench Verified.
Since CHIP and TROLL share the training objective and differ in the projection problem, these results show that replacing the KL projection with the χ² projection improves performance in practice, consistent with Proposition 2.