acceptodds
Under review as a conference paper at ICLR 2027

CHIP: One Chi-Squared Ball for Both Errors of Reinforcement Learning in Large Language Models

10 views

Abstract

Reinforcement learning updates a large language model with data written by the model's previous version, and the error of one such update splits into exactly two terms: a bias, because the objective we can compute scores the new policy on the states the old policy visited, and a variance, because that objective is estimated from a finite batch. We find that the divergence between the new and old policies controls both errors. For ideal importance weights, equals their variance, which governs finite-batch estimation noise; the same budget also bounds the total-variation distance in the bias bound. We enforce this single budget at every sampled position through a projection layer that replaces the actor's proposed next-token distribution with the closest distribution inside the ball around the old policy. The solution has a normalized weighted harmonic form governed by one scalar, and implicit differentiation gives its gradients in closed form. The layer adds less than 1% to each training step and replaces clipping at the same point in standard training recipes. We call this algorithm CHIP (CHi-squared Projections). Replacing clipping with CHIP raises validation accuracy in all five estimator families we tested, by +0.7 to +10.7 points. In a six-scale Qwen3 ladder, CHIP leads in seventeen of eighteen cells and achieves an average validation gain of +6.9 points over the last sixty steps. On multi-turn agentic coding tasks, CHIP achieves 42.8% pass@1 on SWE-bench Verified at training step 150, outperforming CLIP by 5.8 percentage points.

Primary area:
reinforcement learning
Keywords:
large language models, reinforcement learning, policy optimization, chi-squared divergence, trust regions

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PMw3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper splits the error of an off-policy RL update for LLMs into two parts: bias from scoring the new policy on prefixes written by the old one, and variance from estimating it on a finite batch. They argue that the χ² divergence between the new and old policy controls both, while KL does not. Based on this they propose CHIP, which replaces PPO clipping with a projection: whenever a next-token distribution goes past a χ² limit from the old policy, it is moved to the closest distribution within that limit, and a KL term pulls the actor toward this projected distribution. The authors claim the experiment results on math RL show CHIP method outperforms clipping.

Critical strengths
  1. Writing and framing are good. Splitting the update error into these two layers and showing that one quantity covers both is clean, and the paper explains it well. The example with a 1e-6 probability token makes it easy to see why a KL budget can miss what χ² catches.

  2. The method is interesting. It only replaces the clipping step without affecting the advantages of the classic RL algorithm, and the cost is low.

Critical weaknesses
  1. The closest prior work is TROLL (Becker et al., 2026), which already replaces clipping with a differentiable trust region projection on a sparse set of tokens and trains the actor toward the projected distribution. As far as I can see CHIP keeps the same recipe and swaps KL for χ². But TROLL is only cited in passing in Sec 3.4 and is not compared against.

  2. Most of the gains come from runs where GRPO peaks and then falls apart (at 4B, it loses about half its accuracy). At 32B, where GRPO trains fine, the gain is around one point. Every setting is run once, and ε is also changed for the larger models. It can’t tell if CHIP performs better.

Questions for authors
  1. Can you compare against TROLL? It already does the same kind of projection with KL, so this is the most direct way to show that χ² is actually better than KL.

  2. Can you compare against a GRPO run that doesn’t collapse? Right now most of the gain seems to come from GRPO falling apart, so I’d like to see if CHIP still helps when GRPO trains normally.

5 comments

Response to Reviewer PMw3 (1/3): Q1 - Relationship to TROLL

Response to Reviewer PMw3

Revised manuscript. We have uploaded a revised PDF. The changes relevant to this review are: (1) in response to the reviewer's point that TROLL was only cited in passing, a new Related Work paragraph, Trust-region projections (Section 5), that discusses TROLL directly, including the shared training objective, the KL versus χ² projection problems, and their consequence for importance-weight variance; and (2) a new Appendix I.4 that reports the SWE-bench Verified comparison with TROLL and the CLIP + auxiliary-KL control from this response.

Weakness 1 and Question 1: Relationship to TROLL

Thank you for raising this question. The reviewer is right that CHIP keeps TROLL's training objective; we made this choice deliberately and changed only the projection problem. Eq. (8) of CHIP has the same form as Eq. (5) of TROLL: both use the projected distribution in the importance ratio and regress the raw actor toward it with a stop-gradient KL term. Keeping this part fixed isolates the one component our theory is about, the divergence that defines the projection.

1 The projection problems differ

Write πθ\pi_\theta for the raw actor output, πˉθ\bar\pi_\theta for the projected distribution, r=πθ/πoldr=\pi_\theta/\pi_{\mathrm{old}} for the raw ratio, and rˉ=πˉθ/πold\bar r=\bar\pi_\theta/\pi_{\mathrm{old}} for the projected ratio:

TROLL, Eq. (3)CHIP, Eq. (6)
Projection problemarg⁡min⁡pKL(p∥πθ)\arg\min_p \mathrm{KL}(p\Vert\pi_\theta) s.t. KL(p∥πold)≤ε\mathrm{KL}(p\Vert\pi_{\mathrm{old}})\le\varepsilonarg⁡min⁡pχ2(p∥πθ)\arg\min_p \chi^2(p\Vert\pi_\theta) s.t. χ2(p∥πold)≤ε\chi^2(p\Vert\pi_{\mathrm{old}})\le\varepsilon
Closed-form solutionπˉθ∝πoldη/(1+η) πθ1/(1+η)\bar\pi_\theta\propto\pi_{\mathrm{old}}^{\eta/(1+\eta)}\,\pi_\theta^{1/(1+\eta)} (weighted geometric mean)1/πˉθ∝1/πθ+η/πold1/\bar\pi_\theta\propto 1/\pi_\theta+\eta/\pi_{\mathrm{old}} (weighted harmonic mean)
Projected ratiorˉ∝r1/(1+η)\bar r\propto r^{1/(1+\eta)}, unbounded in rr for any fixed η\etarˉ=r/(Z(1+ηr))<1/(Zη)\bar r=r/\big(Z(1+\eta r)\big)<1/(Z\eta), bounded
Importance-weight varianceNo bound from the KL budget (Proposition below)Var⁡πold(rˉ)=χ2(πˉθ∥πold)≤ε\operatorname{Var}_{\pi_{\mathrm{old}}}(\bar r)=\chi^2(\bar\pi_\theta\Vert\pi_{\mathrm{old}})\le\varepsilon

TROLL's geometric solution passes a large raw ratio through as a large projected ratio, while CHIP's harmonic solution caps every projected ratio and bounds their variance by the budget.

2 Why the divergence matters

Proposition 2 (KL blind spot). For every ε>0\varepsilon>0 and every M>0M>0, there exist policies π\pi and πold\pi_{\mathrm{old}} such that KL(π∥πold)≤ε\mathrm{KL}(\pi\Vert\pi_{\mathrm{old}})\le\varepsilon while Var⁡πold(π/πold)=χ2(π∥πold)≥M\operatorname{Var}_{\pi_{\mathrm{old}}}(\pi/\pi_{\mathrm{old}})=\chi^2(\pi\Vert\pi_{\mathrm{old}})\ge M.

A KL ball therefore contains updates with arbitrarily large importance-weight variance, and this variance drives the finite-batch estimation error (Layer 2). Under the stated assumptions, a χ² budget bounds this variance and also bounds the total-variation distance that controls the state-distribution-shift error (Layer 1). The χ² budget controls both error layers, whereas a KL budget controls only Layer 1.

3 Experimental comparison with TROLL

For both Qwen3-14B mathematical reasoning and Qwen3-32B on SWE-bench Verified, CHIP uses a χ² radius of ε=0.25\varepsilon=0.25 and an auxiliary KL coefficient of α=1\alpha=1. TROLL uses the published defaults in Table 3: a KL radius of ε=0.05\varepsilon=0.05 and a KL regression coefficient of α=1\alpha=1. We compared results at common evaluation steps.

The table reports the last four evaluations of each task:

TaskStepTROLLCHIPCHIP gain (pp)
Qwen3-14B math (DAPO validation mean)27060.0%61.7%+1.7
Qwen3-14B math (DAPO validation mean)28059.6%61.3%+1.7
Qwen3-14B math (DAPO validation mean)29060.1%62.1%+2.0
Qwen3-14B math (DAPO validation mean)30060.2%62.4%+2.2
Qwen3-32B SWE-bench Verified (pass@1)10039.0%45.8%+6.8
Qwen3-32B SWE-bench Verified (pass@1)11040.6%41.8%+1.2
Qwen3-32B SWE-bench Verified (pass@1)12039.6%45.4%+5.8
Qwen3-32B SWE-bench Verified (pass@1)13040.4%45.8%+5.4

Over the full runs, CHIP leads TROLL at all 30 evaluations on mathematical reasoning (steps 10–300) and at 10 of the 14 evaluations on SWE-bench Verified (steps 0–130), remaining ahead from step 40 onward. Over the last four evaluations, the mean gain is 1.9 points on mathematical reasoning and 4.8 points on SWE-bench Verified.

Since CHIP and TROLL share the training objective and differ in the projection problem, these results show that replacing the KL projection with the χ² projection improves performance in practice, consistent with Proposition 2.

Response to Reviewer PMw3 (2/3): Weakness 2 and Q2 - Stable GRPO controls

Weakness 2 and Question 2: Does CHIP's advantage depend on collapse of the GRPO baseline?

Thank you for highlighting the importance of baseline stability. CHIP's performance advantage does not require late-training collapse of GRPO. In the original 4B and 14B mathematical-reasoning settings, we continued to observe late-training instability after changing the training seed. To compare against GRPO that trains stably, we increased its reference-policy KL penalty coefficient from 0.001 to 0.003 and conducted additional experiments at these two model scales.

With the stronger KL penalty, neither GRPO run exhibits the late-training collapse seen previously, yet CHIP outperforms GRPO at all 30 common evaluation points in each comparison. We report every evaluation from steps 10 to 300 below. The metric is DAPO validation-suite mean accuracy.

Qwen3-4B mathematical reasoning

StepGRPO (%)CHIP (%)CHIP gain (pp)
1043.949.0+5.1
2048.651.6+3.0
3049.255.4+6.2
4049.455.2+5.8
5052.554.2+1.7
6052.253.2+1.0
7052.254.8+2.6
8050.754.9+4.2
9051.855.2+3.4
10052.256.5+4.3
11052.156.3+4.2
12052.255.7+3.5
13052.755.7+3.0
14052.556.8+4.3
15052.556.6+4.1
16052.854.6+1.8
17054.154.7+0.6
18053.956.5+2.6
19053.954.1+0.2
20052.655.5+2.9
21053.456.5+3.1
22052.856.7+3.9
23052.556.5+4.0
24052.456.5+4.1
25052.755.4+2.7
26054.455.0+0.6
27054.257.0+2.8
28054.154.8+0.7
29054.156.0+1.9
30052.757.5+4.8

Qwen3-14B mathematical reasoning

StepGRPO (%)CHIP (%)CHIP gain (pp)
1044.353.8+9.5
2051.054.4+3.4
3051.755.8+4.1
4052.157.0+4.9
5052.757.6+4.9
6052.557.8+5.3
7055.559.0+3.5
8055.258.5+3.3
9055.659.7+4.1
10054.560.0+5.5
11054.960.3+5.4
12056.259.7+3.5
13056.959.1+2.2
14057.861.5+3.7
15057.661.1+3.5
16057.362.0+4.7
17058.861.8+3.0
18058.660.5+1.9
19057.661.3+3.7
20058.861.7+2.9
21058.059.7+1.7
22059.161.2+2.1
23057.261.7+4.5
24057.559.8+2.3
25059.661.4+1.8
26057.961.9+4.0
27057.661.7+4.1
28059.361.3+2.0
29059.462.1+2.7
30059.662.4+2.8

At step 300, CHIP achieves 57.5% and 62.4% on 4B and 14B, respectively, exceeding GRPO with the stronger KL penalty by 4.8 and 2.8 percentage points. Across the seven evaluations from steps 240 to 300, CHIP's mean advantages are approximately 2.5 and 2.8 percentage points, respectively. The gains therefore persist during training rather than appearing only at a few checkpoints after baseline collapse.

Response to Reviewer PMw3 (3/3): Weakness 2 - SWE, 32B, single runs, and radius

SWE-bench Verified

Figure 2 in Section 6.3 provides further evidence: the CLIP baseline trains without pronounced late-training collapse, while CHIP still achieves better performance. This baseline uses asynchronous CLIP based on the DeepSWE recipe. CHIP remains ahead at every evaluation from step 40 onward; the table reports the last four evaluations (pass@1).

StepCLIP (%)CHIP (%)CHIP gain (pp)
10038.845.8+7.0
11038.841.8+3.0
12038.845.4+6.6
13039.045.8+6.8

At step 130, CHIP achieves 45.8%, compared with 39.0% for CLIP, a gain of 6.8 percentage points; the mean gain over the last four evaluations is 5.9 points.

These results show that CHIP's advantage is present when the baseline trains stably, not only when it collapses. This is consistent with our theoretical analysis: under the stated assumptions, a χ² constraint controls both state-distribution shift error and finite-sample estimation error. Its role is therefore not limited to preventing collapse; the analysis also provides a theoretical basis for sustained performance gains during stable training.

32B, where GRPO trains stably

At 32B, CHIP's gain on the validation-suite mean is smaller, +1.2 points over the final sixty steps, but consistent: CHIP is higher at all 30 synchronized evaluations (Figure 1). On held-out AIME'25, CHIP gains +6.2 points, and DAPO training success increases by +2.4 points (Table 1). The SWE-bench Verified experiment above also uses Qwen3-32B with a stable CLIP baseline, and there CHIP leads by 6.8 points at step 130.

Single training runs

Each comparison is a separately trained matched pair, and the direction of the result is the same across the six model scales in Table 1, the seven algorithm configurations in Table 2, and the new TROLL, auxiliary-KL, and stable-GRPO controls. Within each pair, the gain is not driven by a single checkpoint: across the six scales in Table 1, CHIP is higher at 172 of 177 synchronized evaluations. The baseline collapse is also not a property of one seed: as noted above, rerunning GRPO at 4B and 14B with a different seed reproduced the late-training instability.

Choice of radius

The 0.6B–8B runs use ε=0.05, which is below the best radius at 8B (Table 3 and the ε=0.5 ablation in our response to Reviewer uPU9), so the gains at these scales were obtained without tuning ε to its best value. For 14B and 32B, we tried several radii in preliminary runs and found ε=0.25 to work well, consistent with the 8B ablation. In Table 2, both radii also improve over the clipped baseline for GSPO (+0.7 and +0.9 points) and DAPO (+3.8 and +6.7 points), so the gains do not depend on one particular radius.

Reviewer_uPU9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CHIP, a replacement for PPO-style ratio clipping in LLM reinforcement learning based on a χ2\chi^2-constrained policy projection. Its motivation connects χ2\chi^2 divergence to two sources of error: surrogate approximation bias and finite-batch importance-sampling variance. CHIP projects each token distribution onto a χ2\chi^2 ball around the old policy. The projection has a normalized harmonic form, computed through a one-dimensional root search, with an analytic backward pass and a sparse implementation for large vocabularies. Experiments across multiple Qwen3 model sizes and policy-optimization algorithms evaluate mathematical reasoning, code generation, and SWE-bench Verified, reporting improvements in performance and training stability over clipping-based baselines.

Critical strengths
  1. Practical implementation of a distribution-level constraint. The paper suggests a implementable algorithm for the theory-motivated projection, and the analytic backward pass and sparse representation make it practical for large vocabularies. This method is readily integrable into existing LLM RL training recipes.

  2. Breadth of empirical evaluation. The experiments cover six Qwen3 model scales, five policy-optimization algorithms, and both reasoning and coding tasks. The reported improvements, including a 6.8-percentage-point gain over the clipped control on SWE-bench Verified, suggest that the method could have practical value across different training settings.

Critical weaknesses
  1. Serious mismatch between theory and empirical setup. Theorem 1 assumes γ<1\gamma < 1, whereas episodic LLM RL typically uses an undiscounted objective (γ=1\gamma = 1). The stated bound does not apply to this setting. Consequently, the central claim that one χ2\chi^2 constraint controls both errors is not established for the target setting.

  2. Uncertainty across different runs. The reported standard deviations describe variation across checkpoints, rather than variability across independent training seeds. Without repeated training runs, it is difficult to assess the robustness of the reported improvements, particularly the smaller gains.

Questions for authors
  1. What advantages does the analytic backward pass in Proposition 3 offer over implementing the same projection using existing implicit-differentiation packages?

  2. In the radius ablation, ε=0.25\varepsilon = 0.25, the largest value tested, performs best. Have the authors evaluated larger radii? Where does performance begin to deteriorate, and what evidence supports the selected radius as a useful balance between constraining updates and allowing policy improvement?

  3. The authors provide a counterexample where KL divergence may be suboptimal in theory, and cite a related work[1] that uses KL divergence instaed of χ2\chi^2 as the regularizer. Does this observation extend to practice? That is, how does CHIP compare to TROLL in terms of performance?

[1] Philipp Becker, Niklas Freymuth, Serge Thilges, Fabian Otto, and Gerhard Neumann. TROLL: Trust Regions improve Reinforcement Learning for Large Language Models. International Conference on Learning Representations (ICLR), 2026.

4 comments

Response to Reviewer uPU9 (1/2)

Response to Reviewer uPU9

We have uploaded a revised manuscript; the changes relevant to this review are noted in the answers below.

1 Discounted theory and undiscounted experiments

Thank you for pointing this out. Theorem 1 as originally cited concerns infinite-horizon discounted returns and cannot be evaluated by directly setting γ=1\gamma=1. To address finite-length, undiscounted tasks, we have added a complete analysis and proof in Appendix B.1, pages 16–18 of the revised manuscript, with a pointer in Section 2.2, page 3. The new result is Theorem 3, Eqs. (10)–(13).

LLM generation can be modeled as a finite-horizon process with at most HH decision steps; early termination is represented by a zero-reward absorbing state. The finite-horizon performance-difference lemma involves a finite sum and does not require γ<1\gamma<1. The same two-layer decomposition therefore applies: one error arises from replacing the candidate policy's state distribution with the behavior policy's distribution, and the other arises from estimating the population surrogate with a finite sample.

Let ΔJH=JH(π)−JH(πold)\Delta J_H=J_H(\pi)-J_H(\pi_{\mathrm{old}}) denote improvement in undiscounted total return, and let LHL_H be the time-summed surrogate under the old state distributions. For a fixed candidate policy, assume true advantages bounded by ∣At∣≤B|A_t|\le B, a shared initial distribution and environment, the required behavior-policy support, and χ2(πt∥πold,t)≤ε\chi^2(\pi_t\|\pi_{\mathrm{old},t})\le\varepsilon at every relevant time and state. The new theorem gives

∣ΔJH−LH∣≤B2H(H−1)ε.|\Delta J_H-L_H| \le \frac{B}{2}H(H-1)\varepsilon.

For an unbiased estimate L^H\widehat L_H constructed from NN independent behavior-policy trajectories,

Var⁡(L^H)≤H2B2(1+ε)N.\operatorname{Var}(\widehat L_H) \le \frac{H^2B^2(1+\varepsilon)}{N}.

Combining the two yields a mean-square bound on the total objective error:

E[(ΔJH−L^H)2]≤[B2H(H−1)ε]2+H2B2(1+ε)N.\mathbb{E}[(\Delta J_H-\widehat L_H)^2] \le \left[\frac{B}{2}H(H-1)\varepsilon\right]^2 +\frac{H^2B^2(1+\varepsilon)}{N}.

Here, NN counts independent trajectories, not tokens; the proof allows dependence between tokens within a trajectory. The same χ2\chi^2 budget thus controls the quantities underlying both error layers, yielding a total objective-error bound in the finite-horizon setting with γ=1\gamma=1. What changes is the horizon dependence, not the mechanism of joint control.

2 Uncertainty across checkpoints and training seeds

Thank you for highlighting this point. The reported standard deviations describe variation in performance differences across late-training checkpoints within a training run. Our experiments also show that CHIP achieves higher DAPO validation-suite mean accuracy than the corresponding baseline across all six model sizes and all seven configurations spanning five algorithms, demonstrating consistent gains across multiple settings. Given the computational cost of large-model RL training and the time available for experiments, we leave systematic multi-seed replications under the same configuration to future work.

Question 1: Advantages of the analytic backward pass

Thank you for this question. Existing implicit-differentiation tools can indeed compute the same gradient. The advantage of Proposition 3 is that it exploits the special structure of CHIP's projection to reduce implicit differentiation to an explicit backward formula. The projected distribution has an explicit harmonic form, with only one scalar determined by numerical root-finding in the forward pass. Exploiting this structure, we derive the vector–Jacobian product needed for backpropagation directly, without constructing the full Jacobian or solving the multidimensional derivative system associated with the unreduced constrained problem.

Concretely, the backward pass uses only elementwise operations and a few sums over the kk retained tokens. Both computation and intermediate storage are O(k)O(k) per position, and these operations can be batched directly. The formula also accounts for how the scalar changes with the actor probabilities, without storing or unrolling the forward root-finding iterations. These properties make the projection practical at many token positions during LLM training, with explicit computational cost and straightforward batching. In our implementation, the full projection layer, including this backward pass, adds less than 1% to each training step.

Question 2: Evaluation of a larger radius

Thank you for this question. We also evaluated the larger radius ε=0.5\varepsilon=0.5 and compared it with ε=0.25\varepsilon=0.25 on the Qwen3-8B DAPO mathematical reasoning task. The ε=0.5\varepsilon=0.5 run was trained through step 150, so the table uses their common evaluation interval, steps 10–150, with evaluations every 10 steps.

The metric is DAPO validation accuracy. Differences are the results for ε=0.25\varepsilon=0.25 minus those for ε=0.5\varepsilon=0.5.

Response to Reviewer uPU9 (2/2)

Question 2 (continued): Radius results

The table shows the last four evaluations of the common interval.

Stepε=0.25ε=0.5Difference (pp)
12070.9%69.0%+1.9
13072.8%68.2%+4.6
14071.7%68.7%+3.0
15072.0%68.8%+3.2

ε=0.25\varepsilon=0.25 achieves higher accuracy at all 15 common evaluation points, with an average advantage of 4.4 percentage points and an advantage of 3.2 percentage points at step 150. In this experiment, increasing the radius from 0.25 to 0.5 reduced validation performance.

The radius controls the allowed policy displacement. A smaller radius enforces a tighter constraint but may restrict policy improvement; a larger radius permits freer updates while relaxing the upper bound on importance-weight variance. Together with the smaller-radius ablations in the main paper, these results support choosing 0.25 for this task as a balance between constraining updates and allowing policy improvement: some relaxation helps learning, but increasing the radius further to 0.5 did not improve performance.

This also bears on the radius used in Table 1. The 0.6B–8B runs use ε=0.05\varepsilon=0.05, which is below the best-performing radius at 8B in both Table 3 and this ablation. The gains at those scales were therefore obtained without tuning the radius to its best value for each model.

Question 3: Empirical performance relative to TROLL

Thank you for this question. To evaluate whether the theoretical distinction translates into practical performance gains, we conducted additional experiments directly comparing CHIP with TROLL on Qwen3-14B mathematical reasoning using DAPO and on SWE-bench Verified.

For both Qwen3-14B mathematical reasoning and Qwen3-32B on SWE-bench Verified, CHIP uses a χ² radius of ε=0.25\varepsilon=0.25 and an auxiliary KL coefficient of α=1\alpha=1. TROLL uses the published defaults in Table 3: a KL radius of ε=0.05\varepsilon=0.05 and a KL regression coefficient of α=1\alpha=1. We compared results at common evaluation steps.

The revised manuscript adds a Related Work paragraph on TROLL (Section 5) and reports the SWE-bench Verified comparison in Appendix I.4. The table reports the last four evaluations of each task:

TaskStepTROLLCHIPCHIP gain (pp)
Qwen3-14B math (DAPO validation mean)27060.0%61.7%+1.7
Qwen3-14B math (DAPO validation mean)28059.6%61.3%+1.7
Qwen3-14B math (DAPO validation mean)29060.1%62.1%+2.0
Qwen3-14B math (DAPO validation mean)30060.2%62.4%+2.2
Qwen3-32B SWE-bench Verified (pass@1)10039.0%45.8%+6.8
Qwen3-32B SWE-bench Verified (pass@1)11040.6%41.8%+1.2
Qwen3-32B SWE-bench Verified (pass@1)12039.6%45.4%+5.8
Qwen3-32B SWE-bench Verified (pass@1)13040.4%45.8%+5.4

Over the full runs, CHIP leads TROLL at all 30 evaluations on mathematical reasoning (steps 10–300) and at 10 of the 14 evaluations on SWE-bench Verified (steps 0–130), remaining ahead from step 40 onward. Over the last four evaluations, the mean gain is 1.9 points on mathematical reasoning and 4.8 points on SWE-bench Verified.

CHIP deliberately keeps TROLL's training objective and changes the projection problem from KL to χ². These comparisons therefore show that the χ² projection improves performance in practice, consistent with the KL counterexample in Proposition 2.

Thank you for your rebuttal.

1. Experiments\textbf{1. Experiments} I appreciate the authors' extensive extra experiments, and most of my concerns on the empirical aspects of the paper are resolved. I also understand that running multiple-seed experiments is challenging, especially for these kinds of large-scale LLM experiments.

2. Theory\textbf{2. Theory} I have read the undiscounted version of the new theorem. First of all, I feel like some notation changes made in the appendix, e.g. μ=πold\mu = \pi_{\mathrm{old}} is unnecessary.

And I think the applicability of the theory is limited. The new theorem's bias guarantee is O(H^2 \varepsilon), where H is the horizon. Since H is large for LLM sequences, this bound could be easily vacuous in practical setups. I understand that connecting theory could be challenging in this field, but considering that the proof techniques used are all pretty standard, I am concerned with the theoretical contributions that this paper is making.

Still, since I believe that the main contribution of this paper is empirical, I will maintain my initial score.

Thank you for your careful reassessment and for recognizing the additional experiments. We appreciate your feedback on the horizon dependence and notation. Our analysis is intended to guide the choice of constraint for controlling the two error sources, and we take your point that the worst-case bound can be loose for long sequences. Thank you for your constructive feedback and the time you have devoted to our paper.

Reviewer_Uch6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CHIP an interesting alternative to the standard clipping technique in many RL algorithms and shows that CHIP can be ported into GRPO, PPO, TRPO and shows that CHIP improved the performances of these standard algorithms. CHIP is principled based on Schulman et.al. 2015 and Lemma 1 and Prop 1 of this paper. However, overlaps with prior works needs clarifying.

Critical strengths
  1. Theoretically principled algorithm justified by Schulman et.al. 2015 and Lemma 1 and Prop 1. Make sense that chi square controls both bias and variance.
  2. SWE experiment shows convincing gain against the standard clip trick in RL algorithms, an table 1 shows CHIP scales to different model sizes.
Critical weaknesses
  1. similarity with prior work needs clarifying
  2. effect of the projection layers and kl stabilizer not isolated
Questions for authors
  1. What is the difference of this work with TROLL. TROLL's objective equation 5 looks similar to CHIP's objective equation 8. Seems like TROLL has done similar benchmark in its table 1 similar to CHIP's table 1. I m willing to raise my score if this is clarified.

    Becker, P., Freymuth, N., Thilges, S., Otto, F., and Neumann, G. TROLL: Trust Regions improve Reinforcement Learning for Large Language Models. In International Conference on Learning Representations (ICLR), 2026. arXiv:2510.03817. https://arxiv.org/abs/2510.03817

  2. Is the KL stabilizer here doing the heavy lifting in objective 8, rather than the projected distribution πˉθ\bar\pi_{\theta}. Authors can include a short ablation of CLIP with the KL stabilizer to isolate effects.

7 comments

Response to Reviewer Uch6 (1/4): Q1 - Same objective, different projection problem

Response to Reviewer Uch6

Revised manuscript. We have uploaded a revised PDF. The changes relevant to this review are: (1) a new Related Work paragraph, Trust-region projections (Section 5), that discusses TROLL directly, including the shared training objective, the KL versus χ² projection problems, and their consequence for importance-weight variance; and (2) a new Appendix I.4 that reports the SWE-bench Verified comparison with TROLL and the CLIP + auxiliary-KL control from this response.

Question 1: Relationship to TROLL

Thank you for raising this question. CHIP deliberately keeps TROLL's training objective and changes the projection problem. Eq. (8) of CHIP has the same form as Eq. (5) of TROLL. We made this choice so that the comparison isolates one component: the divergence that defines the projection.

Regarding the similarity of the benchmarks in Table 1: both papers evaluate on standard mathematical-reasoning benchmarks, which are the common testbeds for RL training of LLMs. CHIP additionally evaluates multi-turn agentic software engineering on SWE-bench Verified (Section 6.3), and the new experiments in Section 5 of the next comment compare CHIP and TROLL directly on both task types.

That component is what our theory is about. Starting from the performance-difference lemma, we decomposed the error of one update into a state-distribution-shift error (Layer 1) and a finite-batch estimation error (Layer 2). Under the stated assumptions, a single χ² budget controls both layers, whereas a KL budget controls only Layer 1. This analysis determined χ² as the quantity to constrain; the projection is our means of enforcing it.

1 The training objectives coincide

We write both methods in one notation: πθ\pi_\theta is the raw actor output, πˉθ\bar\pi_\theta is the projected distribution, and ⌊⋅⌋\lfloor\cdot\rfloor denotes stop-gradient. (TROLL writes π~θ\tilde\pi_\theta for the raw output and πθ\pi_\theta for the projection.) Both Eq. (5) of TROLL and Eq. (8) of CHIP read

J(θ)=E[1∣o∣∑t(πˉθ(ot∣st)πold(ot∣st)A^t−α KL(πθ(⋅∣st) ∥ ⌊πˉθ(⋅∣st)⌋))].J(\theta)=\mathbb{E}\left[\frac{1}{|o|}\sum_t\left(\frac{\bar\pi_\theta(o_t\mid s_t)}{\pi_{\mathrm{old}}(o_t\mid s_t)}\hat A_t-\alpha\,\mathrm{KL}\left(\pi_\theta(\cdot\mid s_t)\,\Vert\,\lfloor\bar\pi_\theta(\cdot\mid s_t)\rfloor\right)\right)\right].

The objective therefore cannot distinguish the two methods. The difference lies entirely in how πˉθ\bar\pi_\theta is computed.

2 The projection problems differ

TROLL's Eq. (3) and CHIP's Eq. (6) define πˉθ\bar\pi_\theta as follows, with r=πθ/πoldr=\pi_\theta/\pi_{\mathrm{old}} the raw ratio and rˉ=πˉθ/πold\bar r=\bar\pi_\theta/\pi_{\mathrm{old}} the projected ratio:

TROLL, Eq. (3)CHIP, Eq. (6)
Projection problemarg⁡min⁡pKL(p∥πθ)\arg\min_p \mathrm{KL}(p\Vert\pi_\theta) s.t. KL(p∥πold)≤ε\mathrm{KL}(p\Vert\pi_{\mathrm{old}})\le\varepsilonarg⁡min⁡pχ2(p∥πθ)\arg\min_p \chi^2(p\Vert\pi_\theta) s.t. χ2(p∥πold)≤ε\chi^2(p\Vert\pi_{\mathrm{old}})\le\varepsilon
Closed-form solutionπˉθ∝πoldη/(1+η) πθ1/(1+η)\bar\pi_\theta\propto\pi_{\mathrm{old}}^{\eta/(1+\eta)}\,\pi_\theta^{1/(1+\eta)} (weighted geometric mean)1/πˉθ∝1/πθ+η/πold1/\bar\pi_\theta\propto 1/\pi_\theta+\eta/\pi_{\mathrm{old}} (weighted harmonic mean)
Projected ratiorˉ∝r1/(1+η)\bar r\propto r^{1/(1+\eta)}, unbounded in rr for any fixed η\etarˉ=r/(Z(1+ηr))<1/(Zη)\bar r=r/\big(Z(1+\eta r)\big)<1/(Z\eta), bounded
Importance-weight varianceNo bound from the KL budget (Proposition below)Var⁡πold(rˉ)=χ2(πˉθ∥πold)≤ε\operatorname{Var}_{\pi_{\mathrm{old}}}(\bar r)=\chi^2(\bar\pi_\theta\Vert\pi_{\mathrm{old}})\le\varepsilon

The last two rows are the consequence of the choice. TROLL's geometric solution passes a large raw ratio through as a large projected ratio, while CHIP's harmonic solution caps every projected ratio. Moreover, when the raw output already lies inside the KL ball, TROLL returns it unchanged. The proposition in the next comment shows that the KL ball contains distributions with arbitrarily large importance-weight variance, so TROLL's projected distribution can carry such variance into the RL term of the objective. CHIP's projected distribution satisfies the variance bound by construction.

Response to Reviewer Uch6 (2/4): Q1 - Why KL cannot bound the variance, and TROLL results

3 Why a KL constraint alone cannot control Layer 2 sampling variance

Proposition. For every ε>0\varepsilon>0 and every M>0M>0, there exist policies π\pi and πold\pi_{\mathrm{old}} on an alphabet containing at least two tokens such that

KL(π∥πold)≤ε,Var⁡πold(ππold)=χ2(π∥πold)≥M.\mathrm{KL}(\pi\Vert\pi_{\mathrm{old}})\le\varepsilon, \qquad \operatorname{Var}_{\pi_{\mathrm{old}}} \left(\frac{\pi}{\pi_{\mathrm{old}}}\right) =\chi^2(\pi\Vert\pi_{\mathrm{old}})\ge M.

In plain terms, even a very small KL budget can admit updates with highly variable importance weights. Such updates can also produce large sampling fluctuations in finite-batch estimates. Thus, tightening the KL budget alone does not guarantee control of Layer 2 sampling variance. Under the stated assumptions, a χ² budget provides an explicit bound on this variance while also controlling Layer 1 distribution-shift error. This is the core reason for our choice of χ².

4 Error control provided by the two constraints

Layer 1 arises because the new and old policies visit different state distributions. Layer 2 arises from replacing a population expectation with one finite batch. Even for the same candidate policy, different batches produce different objective estimates. Thus, the quantity of interest in Layer 2 is the sampling variance of the finite-batch estimator; importance-weight variance is an intermediate quantity used to derive its bound. Under the paper's theoretical assumptions, the two divergence budgets provide the following control:

Quantity to controlTROLL: KL budgetCHIP: χ² budget
Layer 1: state-distribution-shift errorControlled through a bound on total variationControlled through a bound on total variation
Layer 2: sampling variance of the finite-batch estimatorNo uniform variance bound from the KL budget aloneThe χ² budget bounds estimator variance through the second moment of the importance weights

The global Layer 1 bound requires the constraint to hold at all relevant states. The Layer 2 variance bound assumes a fixed candidate policy, the corresponding independent sampling scheme, and bounded advantages. In practice, CHIP applies projection at the sampled positions entering the training objective. The key distinction is that the χ² budget provides the control needed for both error layers, whereas a KL budget alone provides only the Layer 1 control.

5 Empirical comparison with TROLL

We conducted additional experiments directly comparing CHIP with TROLL on Qwen3-14B mathematical reasoning and SWE-bench Verified. For both Qwen3-14B mathematical reasoning and Qwen3-32B on SWE-bench Verified, CHIP uses a χ² radius of ε=0.25\varepsilon=0.25 and an auxiliary KL coefficient of α=1\alpha=1. TROLL uses the published defaults in Table 3: a KL radius of ε=0.05\varepsilon=0.05 and a KL regression coefficient of α=1\alpha=1. The following table reports the final evaluation point for each task:

TaskEvaluation stepMetricTROLLCHIPCHIP gain
Qwen3-14B mathematical reasoning300DAPO validation-suite mean accuracy60.2%62.4%+2.2 pp
SWE-bench Verified130pass@140.4%45.8%+5.4 pp

These results show that CHIP achieves higher task performance than TROLL, which shares its training objective and differs in the projection problem, supporting the practical value of the χ2\chi^2 formulation. The theory and experiments answer complementary questions: the theory explains why even an exact KL constraint is insufficient for joint control of the two error layers, while the experiments show practical gains from the χ2\chi^2-based approach on these two tasks.

Response to Reviewer Uch6 (3/4): Q2 - KL stabilizer ablation

Question 2: KL stabilization and the GRPO baseline

Thank you for raising this question. We first clarify the GRPO configuration in the main experiments, then report a direct ablation of the auxiliary KL term in Eq. (8), followed by the existing clip-only control.

1 The main experiments use GRPO with reference-policy KL

Our GRPO baseline follows the original GRPO formulation and retains its reference-policy KL loss. This term is included in the GRPO objective in DeepSeekMath. The GRPO baseline in Table 1 and Figure 1 uses this configuration. Omitting the details of batch and token averaging, its maximization objective is

JGRPO(θ)=E[min⁡(rt(θ)A^t,clip⁡(rt(θ),1−ϵ,1+ϵ)A^t)]−βEs[DKL(πθ(⋅∣s)∥πref(⋅∣s))],J_{\mathrm{GRPO}}(\theta) = \mathbb{E}\left[ \min\left( r_t(\theta)\hat A_t, \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat A_t \right) \right] - \beta \mathbb{E}_s\left[ D_{\mathrm{KL}}\left(\pi_\theta(\cdot|s) \Vert \pi_{\mathrm{ref}}(\cdot|s)\right) \right],

where rt(θ)=πθ(at∣st)/πold(at∣st)r_t(\theta)=\pi_\theta(a_t|s_t)/\pi_{\mathrm{old}}(a_t|s_t), πold\pi_{\mathrm{old}} is the behavior policy that generated the current batch, and πref\pi_{\mathrm{ref}} is the fixed reference policy. The first term is the clipped policy objective, and the second penalizes deviations from the reference policy, providing an additional stabilization mechanism. Thus, the improvements in the main experiments are obtained against GRPO with both clipping and reference-policy KL regularization.

2 Direct ablation of the KL stabilizer in Eq. (8)

To directly test whether the gains are primarily due to the auxiliary KL term in Eq. (8), we conducted additional experiments with a CLIP + Eq. (8) KL control. Its objective is

JCLIP+auxKL(θ)=E[min⁡(rt(θ)A^t,clip⁡(rt(θ),1−ϵ,1+ϵ)A^t)−αDKL(πθ(⋅∣st)∥sg⁡[πˉθ(⋅∣st)])].J_{\mathrm{CLIP+auxKL}}(\theta) = \mathbb{E}\left[ \min\left( r_t(\theta)\hat A_t, \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat A_t \right) - \alpha D_{\mathrm{KL}}\left( \pi_\theta(\cdot\mid s_t) \middle\Vert \operatorname{sg}\left[\bar\pi_\theta(\cdot\mid s_t)\right] \right) \right].

Here, πˉθ\bar\pi_\theta is the same χ²-projected distribution as in Eq. (8), and sg⁡\operatorname{sg} denotes stop-gradient. The RL term in this control still uses the clipped objective of the raw actor; the projected distribution serves only as the target of the auxiliary KL term. Neither this objective nor Eq. (8) includes reference-policy KL. Comparing this control with full CHIP tests whether using the projected distribution directly in the RL objective provides further gains when the auxiliary KL mechanism is retained.

We conducted this ablation on Qwen3-14B mathematical reasoning and Qwen3-32B on SWE-bench Verified. The table reports the last four evaluations of each task:

TaskStepCLIP + Eq. (8) KLCHIPCHIP gain (pp)
Qwen3-14B math (DAPO validation mean)27056.0%61.7%+5.7
Qwen3-14B math (DAPO validation mean)28055.6%61.3%+5.7
Qwen3-14B math (DAPO validation mean)29055.4%62.1%+6.7
Qwen3-14B math (DAPO validation mean)30055.3%62.4%+7.1
Qwen3-32B SWE-bench Verified (pass@1)10037.6%45.8%+8.2
Qwen3-32B SWE-bench Verified (pass@1)11037.0%41.8%+4.8
Qwen3-32B SWE-bench Verified (pass@1)12037.8%45.4%+7.6
Qwen3-32B SWE-bench Verified (pass@1)13038.2%45.8%+7.6

Over the full runs, CHIP leads at all 30 evaluations on mathematical reasoning (steps 10–300) and at 11 of the 14 evaluations on SWE-bench Verified (steps 0–130), remaining ahead from step 30 onward. Over the last four evaluations, the mean gain is 6.3 points on mathematical reasoning and 7.1 points on SWE-bench Verified. These results show that adding the auxiliary KL term from Eq. (8) to CLIP does not match full CHIP's performance; using the projected distribution directly in the RL objective provides additional gains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.