acceptodds
Under review as a conference paper at ICLR 2027

Coefficient Gradient Impact: Calibrating Executed Trust Regions in LLM RL

Abstract

Trust-region mechanisms are ubiquitous in reinforcement learning for language-model reasoning, yet their control parameters inhabit incompatible geometries and say little about the intervention actually executed by training code. We introduce Coefficient Gradient Impact, an objective-aligned measure of the relative displacement of the token coefficients that form the policy gradient. Our analysis yields exact identities for hard removal and smooth reweighting, separates activation frequency from coefficient energy, and characterizes how coefficient-space interventions propagate to parameter gradients. Building on this measure, we develop a baseline-aware calibration procedure that maps heterogeneous trust-region mechanisms onto a common, baseline-declared reporting coordinate, together with a one-sided impact-capping rule that adapts the control threshold while preserving the underlying objective geometry. An immutable-trace protocol makes the measurement auditable across rollout, anchor, and current policies while preserving scoring and synchronization provenance. Multi-run experiments on physical servers across model scales and numerical precisions indicate that the framework enables calibrated comparison among otherwise incomparable mechanisms, tracks parameter-gradient change more faithfully than the tested alternatives, provides useful early warning of unstable training, and transfers across the tested execution settings. Across paired training runs, impact-guided control exhibits the strongest observed stability profile while maintaining comparable reasoning accuracy. Coefficient Gradient Impact therefore provides a principled interface between trust-region design, systems execution, and empirical evaluation, enabling more interpretable comparison and auditable control of language-model reinforcement learning.

Primary area:
reinforcement learning
Keywords:
large language model reinforcement learning, policy optimization, trust-region methods, gradient diagnostics, calibration, training stability, training-inference mismatch

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_SFS5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Coefficient Gradient Impact (CGI), which measures the relative change that a trust-region mechanism makes to objective-weighted token-gradient coefficients. CGI provides a common reporting scale relative to each mechanism’s declared baseline and supports adaptive threshold adjustment to limit excessive intervention. In the tested settings, it tracks parameter-gradient changes more closely than activation frequency, while CGI-guided control produces fewer observed training collapses with comparable reasoning accuracy. The metric is promising, although stronger comparisons with simpler diagnostics and well-tuned controls are needed to establish its practical advantage.

Critical strengths
  1. A central strength is the proposed CGI metric, which provides a unified and interpretable measure of trust-region intervention strength. By quantifying relative changes in objective-weighted token-gradient coefficients, it offers a coherent basis for comparing heterogeneous mechanisms under explicitly declared baselines. This is consequential because nominal thresholds and activation rates alone do not capture the magnitude of the intervention, making CGI a meaningful contribution to how these methods are evaluated and calibrated.
Critical weaknesses
  1. A central weakness is the lack of targeted ablations isolating the benefits of CGI’s weighting and adaptive control. Comparing CGI with unweighted RMS or mean absolute deviations of the execution weights would clarify whether weighting by baseline coefficient energy provides additional predictive value. For the controller, fixed-threshold comparisons with a clearly matched tuning budget would help distinguish the benefit of per-update adaptation from simply selecting a better regularization strength. Without these controls, it remains unclear which specific design choices drive the reported improvements.
Questions for authors
  1. What motivates CGI’s relative L2 formulation and its implicit weighting by squared baseline coefficients? Would ablations against simpler alternatives, such as unweighted RMS or mean absolute weight deviations, help establish the value of these design choices?
  2. How does CGI correlate with response length, entropy, and changes in these quantities during training? Does it provide additional predictive value beyond these signals, particularly in terms of warning lead time and false-positive rate?
  3. Given the five paired runs, how much uncertainty remains around the observed reduction in collapse, particularly relative to BandPO?
6 comments
AuthorsOpenReview ↗

Official Response to Reviewer SFS5 (Part 1/5)

We thank the reviewer for the careful reading and constructive feedback. We address the questions and clarify the issues below.

About Relative L2 and Coefficient-Energy Weighting

What motivates the relative L2 formulation, and does objective weighting add value over unweighted RMS or mean absolute deviations?

Experimental setting.

Two training roots (4104/4105), 20 snapshots/root at outer steps 0,6,…,1140,6,\ldots,114, eight hash-selected observer responses/snapshot, and the response-mean/token-mean reduction. Fixed controls: ratio-hard 0.013472934190594928; ratio-smooth 0.0828465938568115; TV-hard 0.004293510210083452; TV-smooth 0.027021306037902832. Spearman correlations are computed separately within each root/family/control across calibration and held-out snapshots, with the training root as the independent unit.

Evidence already in the submission (Tables 1 and 7). Held-out CGI values are 0.318/0.292/0.306/0.287 for ratio-hard/ratio-smooth/TV-hard/TV-smooth, with activation fractions of 11.7%/99.6%/6.3%/99.4%. TV-smooth has mean weight 0.991 despite CGI 0.287. These existing measurements show the additional information supplied by objective-weighted intervention magnitude.

For positive baseline coefficient energy, CGI is relative Euclidean displacement of the coefficients that multiply token score vectors. If cmode,t=cbase,twtc_{{\rm mode},t}=c_{{\rm base},t}w_t, then

Ic2=∑tut(wt−1)2∑tut,ut=(qtA^trt)2.I_c^2=\frac{\sum_t u_t(w_t-1)^2}{\sum_t u_t},\qquad u_t=(q_t\hat A_t r_t)^2.

Squared coefficient-energy weighting follows from that estimand. The reduction weights qtq_t include masks and response lengths; zero-advantage coefficients contribute no baseline energy. Shared nonzero coefficient rescaling cancels in the relative normalization (§4.1).

Unweighted RMS corresponds to equal baseline coefficient magnitudes. Mean absolute deviation also changes the norm. Signed mean-weight deviations can cancel, so they do not replace either ablation. Slash-separated cells below list root 4104/root 4105, with within-root correlations rather than a pooled estimate.

Smooth familyFixed control (rounded)CGIUnweighted RMS(w−1)Mean absolute deviation
SPO (ratio)0.0828465940.657/0.6440.346/0.0330.284/0.402
DRPO (Binary-TV)0.0270213060.747/0.7110.370/0.4230.251/0.086

CGI exceeds both unweighted alternatives in all four smooth-family/root comparisons. This empirical comparison tests the predictive value of weighting; the coefficient-energy interpretation follows directly from the definition.

Analytical example. Let one of 100 baseline coefficients have magnitude 10 and the other 99 magnitude 1. Removing any one coefficient gives activation 1%, unweighted RMS 0.1 and mean absolute deviation 0.01. Removing the large coefficient gives CGI 10/199≈0.70910/\sqrt{199}\approx0.709; removing an ordinary coefficient gives 1/199≈0.0711/\sqrt{199}\approx0.071. The tenfold distinction is precisely the coefficient-energy information retained by Eq. 7.

AuthorsOpenReview ↗

Official Response to Reviewer SFS5 (Part 2/5)

About Length, Entropy and Their Changes

How does CGI relate to response length, entropy and changes in these quantities?

Experimental setting.

Roots 4104/4105; SPO control 0.0828465938568115 and DRPO 0.027021306037902832; 20 snapshots/root at 0,6,…,1140,6,\ldots,114; 19 first-difference pairs at six-outer-step intervals. CGI uses the eight-response observer; mean retained response length and response-mean token entropy use the full training population. Spearman correlations are computed separately within each root.

Length and masks enter CGI through the objective reduction. Entropy is not itself a term in Eq. 4, and the definition imposes no universal monotone relationship with either length or entropy. Level correlations and first-difference correlations describe different associations.

FamilyRootρ(CGI,L), n=20ρ(CGI,H), n=20ρ(ΔCGI,ΔL), n=19ρ(ΔCGI,ΔH), n=19
SPO4104-0.1940.1050.0120.232
SPO4105-0.093-0.0870.156-0.337
DRPO4104-0.212-0.0140.147-0.154
DRPO4105-0.0690.325-0.3110.053

The level and first-difference correlations vary across roots and families, indicating that CGI is not simply a monotone proxy for length or entropy in these measurements. CGI uses the observer subset, while length and entropy summarize the full training population. The matched feature ablations in Parts 3–4 directly test incremental prediction.

Analytically, if every used coefficient has wt−1=dw_t-1=d and baseline energy is positive, CGI is ∣d∣|d| regardless of the length/advantage/ratio composition. This separates the coefficient-weight effect from response-length composition. Parts 3–4 directly test incremental drift prediction.

AuthorsOpenReview ↗

Official Response to Reviewer SFS5 (Part 3/5)

About Incremental Next-Inner Drift Prediction

Does adding CGI improve prediction beyond length, entropy and their changes?

Experimental setting.

We use complete profiles from 20 matched-training runs: five roots 6101–6105 × SPO/DRPO × fixed/cap, evaluated at 120 and 240 outer-step horizons. Each case predicts anchor-relative unweighted-token-mean drift D4D_4 before update 4 from inner-3 features in the same outer step. At 120 steps: train outer 0–59, purge 60, test 61–119; at 240: train 0–119, purge 120, test 121–239. All models share valid cases. M0 predicts D3D_3; M1 uses D3D_3, log mean retained length LL, entropy H3H_3, H3−H2H_3-H_2 and the change in log length from the previous outer step (initial difference 0). M2 adds CGI3\mathrm{CGI}_3; M3 adds the preclip gradient norm at inner 3 to M1; M4 adds CGI3\mathrm{CGI}_3 to M3. Ridge uses alpha 1, an unpenalized intercept, training-only mean/std(ddof=0), and constant-column scale 1. Targets retain their original scale and predictions are unclipped. High-drift labels and alarms use the training-target 90th percentile (linear interpolation); the forecast lead is one inner update. We average the four arms within each root and use a five-root paired 10,000-resample bootstrap. Undefined class-based metrics are recorded as null. This drift endpoint is separate from the collapse-onset AUROC in Table 8.

M2 versus M1 and M4 versus M3 add CGI to otherwise matched feature sets on the same temporally separated cases. The endpoint is anchor-relative drift before update 4, predicted using inner-3 features. Table 8 separately reports collapse-onset discrimination.

HorizonModelRMSE x1e3MAE x1e3R2
120M0 persistence0.5020.3070.748
120M1 length/entropy/current drift0.4400.2440.806
120M2 M1 + CGI0.4080.2340.834
120M3 M1 + gradient norm0.4970.3290.753
120M4 M3 + CGI0.2970.1360.912
240M0 persistence0.5460.3470.793
240M1 length/entropy/current drift0.4780.3160.841
240M2 M1 + CGI0.2930.1910.940
240M3 M1 + gradient norm0.5650.2670.778
240M4 M3 + CGI0.5260.1320.808

Calculated before rounding, cohort-mean RMSE decreases for M2 versus M1 by 7.46%/38.61% at 120/240 steps, and for M4 versus M3 by 40.11%/6.82%. These are relative changes in aggregated RMSE. All four comparisons favor adding CGI.

MAE also decreases in all four feature-set/horizon contrasts. The M4-versus-M3 comparison shows lower cohort-mean errors after adding CGI when current drift, length/entropy features and pre-clipping gradient norm are already available.

RMSE and MAE measure prediction error, and R2 measures normalized fit. The feature comparisons use the same fitting procedure and held-out cases, with results aggregated across the four arms within each root.

The two horizons share trajectories and use different fitting/test windows. The forecast lead is fixed at one inner update, and the endpoint is next-inner anchor-relative drift.

Reviewer_XNY9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Coefficient Gradient Impact (CGI), a normalized measure of the change that a trust-region mechanism induces in objective-weighted token coefficients of the policy gradient. The framework covers hard removal and smooth reweighting in both ratio and Binary-TV geometries, while explicitly declaring the reference policy, objective reduction, and token population. The authors derive identities for these mechanisms, describe how coefficient changes are transported into parameter-gradient changes, and propose both a calibration procedure and a one-sided adaptive impact cap. An immutable-trace protocol separates behavior-policy differences, training-anchor differences, and numerical discrepancies between execution engines. Experiments on DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-4B-Base report calibrated intervention strengths, correlations with gradient changes, early-warning performance, and encouraging stability outcomes with comparable mathematical reasoning accuracy.

Critical strengths
  1. They address an interesting point : Nominal thresholds, activation fractions, and average weights are insufficient to compare interventions across different geometries and execution rules. Incorporating advantages, importance ratios, and the actual loss reduction makes the proposed diagnostic more informative than simply counting clipped or affected tokens.

  2. The measurement protocol is useful and technically thoughtful. Distinguishing behavior, anchor, and current policies, and requiring aligned teacher-forced scoring on immutable traces, provides a sound basis for separating numerical mismatch from genuine policy movement.

  3. The work is comprehensive. Five paired runs, held-out calibration snapshots, a gradient-norm-matched arm, an adaptive-bound comparator, and downstream evaluations provide more substantial evidence than a single-run diagnostic study. The reported stability results are promising, particularly because they are accompanied by competitive downstream accuracy.

Critical weaknesses
  1. Table 9 compares the cap with a single fixed threshold. A well-tuned fixed smooth baseline is needed to determine whether the improvement requires adaptive feedback.

  2. On a fixed snapshot, both coefficient and gradient displacement scale with (1/\theta) for smooth methods. Validation should therefore include fixed-threshold comparisons across snapshots and larger observer samples.

  3. Table 8 does not clearly report the prediction horizon, negative-window construction, event counts, or temporal split, making the reported AUROC difficult to assess.

  4. Under AdamW, different moment histories can produce different parameter-step magnitudes despite equal gradient norms. Actual parameter-update norms are needed to support the claimed mechanistic interpretation.

Questions for authors
  • Which smooth objective and numerical loose threshold are used in Table 9, and is the adaptive threshold detached before backpropagation? How frequently does the cap activate?

  • How does the controller compare with a fixed smooth threshold selected under the same calibration budget? What are the collapse outcomes for the calibrated SPO and DRPO baselines already evaluated in Table 11?

  • What are the exact prediction horizon, negative-window sampling procedure, event counts, and temporal split behind the AUROC in Table 8?

  • Does the norm-matched arm achieve comparable actual AdamW parameter-update norms? Does the coefficient-to-gradient relationship remain strong at fixed controls and with larger observer samples?

8 comments
AuthorsOpenReview ↗

Official Response to Reviewer XNY9 (Part 1/7)

We thank the reviewer for the careful reading and constructive feedback. We address the questions and clarify the issues below.

About the Equal-Calibration Fixed Smooth Comparison

How does the controller compare with a fixed smooth threshold selected under the same calibration budget?

Experimental setting.

Five paired roots 6101–6105; SPO/DRPO fixed versus cap; DeepSeek-R1-Distill-Qwen-1.5B on DAPO-Math-17k; 32 prompt groups × 8 responses; four inner updates; BF16 training/rollout; 4,096 response-token and 6,144 context limits; AdamW lr 10−610^{-6}, weight decay 0.01, betas (0.9,0.999), epsilon 10−810^{-8}; clip 1.0. Both arms use the same full-population prepass and calibration budget (16/20 snapshots/root, original grids, target 0.30); SPO loose/fixed 0.10, DRPO 0.03, cap 0.30; endpoints 120/240 outer steps. First onset is evaluated from step 10 after the first-ten-step entropy baseline: reward drop>30% of running max OR entropy outside [0.5, 2] times baseline. Seeds, masks, reduction, threshold selection and the event rule are shared across arms.

Evidence from the submission (Table 11). The paper reports avg@32 accuracy of 31.1/24.2/84.1 for calibrated SPO, 31.8/24.0/84.4 for DRPO, and 31.2/24.1/84.3 for the impact cap on AIME24/AIME25/MATH-498. These results concern the original checkpoint cohort; the additional matched-cohort experiment is reported separately below.

The comparison isolates adding the adaptive operator to the same calibrated smooth objective. Within each family, fixed and capped arms share calibration snapshots, the control grid, selected base threshold, optimizer, prompt stream and response/update budget. Threshold selection minimizes calibration impact error to 0.30 without selecting on reward, accuracy or onset outcomes.

The specified loose/fixed thresholds are SPO ϵ=0.10\epsilon=0.10 and DRPO δ=0.03\delta=0.03. The capped arm uses the same loose value with θt=max⁡(θloose,St/0.30)\theta_t=\max(\theta_{\rm loose},S_t/0.30). This is the one-sided operator of Eq. 10: already-admissible populations remain unchanged.

The results in Part 7 show first-onset events in 5/5 fixed runs versus 2/5 capped runs in each family/horizon. Reward is higher for the cap in DRPO at 120 steps and lower in the other three comparisons. The comparison evaluates the complete operator under the same calibration budget.

The comparison holds the starting calibration and tuning budget fixed. The cap then adapts intervention strength to each population, affecting both the training-average impact and its tail.

AuthorsOpenReview ↗

Official Response to Reviewer XNY9 (Part 2/7)

About Fixed-Control Gradient Correspondence

Smooth coefficient and gradient displacement share inverse-threshold scaling on a fixed snapshot; validation should hold controls fixed across snapshots.

Experimental setting.

Two training roots (4104/4105), 20 snapshots/root at outer steps 0,6,…,1140,6,\ldots,114, eight hash-selected observer responses/snapshot, and the response-mean/token-mean reduction. Fixed controls: ratio-hard 0.013472934190594928; ratio-smooth 0.0828465938568115; TV-hard 0.004293510210083452; TV-smooth 0.027021306037902832. Spearman correlations are computed separately within each root/family/control across calibration and held-out snapshots, with the training root as the independent unit.

We hold the control fixed within each root/family to test correspondence across training states. This separates the across-state relationship from the shared inverse-threshold scaling δc(θ)=v/θ\delta c(\theta)=v/\theta and δg(θ)=Zv/θ\delta g(\theta)=Zv/\theta on a single snapshot. A/B denote roots 4104/4105, respectively; correlations are computed separately within each root.

Family; fixed control (rounded)CGI rho A/BUnweighted RMS(w-1) rho A/BMean abs(w-1) rho A/B
Ratio/hard; 0.0134729340.693/0.6770.268/0.3220.268/0.322
Ratio/smooth; 0.0828465940.657/0.6440.346/0.0330.284/0.402
TV/hard; 0.0042935100.725/0.5800.072/0.3100.072/0.310
TV/smooth; 0.0270213060.747/0.7110.370/0.4230.251/0.086

CGI exceeds both unweighted alternatives in all four smooth-family/root comparisons. These fixed-control results support the predictive value of coefficient weighting across training states. The independent units are the two roots, with snapshots nested within each root.

The favorable ordering also holds in both hard families. It therefore spans hard removal and smooth reweighting at fixed controls, directly addressing the shared inverse-threshold-sweep concern.

For positive baseline coefficient energy, the exact identity is Ic2=∑tcbase,t2(wt−1)2/∑tcbase,t2I_c^2=\sum_t c_{{\rm base},t}^2(w_t-1)^2/\sum_t c_{{\rm base},t}^2, with cbase,t=−qtA^trtc_{{\rm base},t}=-q_t\hat A_t r_t. Whether it predicts a parameter-gradient displacement additionally depends on the score-vector map ZZ (Eq. 8). Thus exact coefficient measurement and setting-dependent predictive performance are separate claims.

For hard removal, w∈{0,1}w\in\{0,1\} gives unweighted RMS =f=\sqrt f and mean absolute deviation =f=f, where ff is the removed fraction. Their equal Spearman correlations follow from this monotone relationship.

About Separating Weighting and Adaptive Feedback

Which controls distinguish objective weighting from adaptive feedback?

Fixed-control comparisons across snapshots remove inverse-threshold sweeps as an explanation of correlation. Comparing CGI with unweighted RMS and mean absolute weight deviation tests the predictive role of objective weighting. Larger nested observers and common-state restriction test sampling sensitivity separately.

For a valid smooth population with positive baseline coefficient energy, Eq. 6 gives Ic=S/θI_c=S/\theta. A fixed threshold that meets a pointwise ceiling on a finite declared set can satisfy Θ≥max⁡(θloose,max⁡tSt/Icap)\Theta\geq\max(\theta_{\rm loose},\max_t S_t/I_{\rm cap}). This is an analytical bound on the declared set. Equation 10 chooses the smallest admissible threshold for each population, allocating intervention according to its current coefficient statistic.

The matched fixed/cap experiment evaluates the complete adaptive operator at the same starting calibration. Equation 10 supplies the per-population ceiling; the training comparison measures the resulting stability and reward outcomes.

AuthorsOpenReview ↗

Official Response to Reviewer XNY9 (Part 3/7)

About Actual AdamW Step Magnitudes

Does gradient-norm matching also yield comparable AdamW parameter-update magnitudes?

Experimental setting.

Two retained final model/AdamW states (roots 4104/4105, next_step=120), immutable step 110 traces, nested hash-ranked observers of eight/32 responses from 256, response mean of token means, and behavior reference μ\mu. DRPO fixed/loose δ=0.027021306037902832\delta=0.027021306037902832, cap 0.30, clipping 1.0; stored BF16 parameters/moments; AdamW lr 10−610^{-6}, weight decay 0.01, betas (0.9,0.999), epsilon 10−810^{-8}. Each arm forks the same serialized state within its comparison.

We directly measure ∥θafter−θbefore∥2\lVert\theta_{\rm after}-\theta_{\rm before}\rVert_2 from stored parameters. Forking identical model and moment states accounts for the AdamW history when comparing actual step magnitudes. A/B denote roots 4104/4105, respectively, with nested observer sizes evaluated within each state.

The step-norm display unit is 10−610^{-6}.

StateResponsesFixedCapFixed, gradient-norm matched to cap
A81521.6021519.9781520.016
A321519.4521518.0841517.970
B81651.6561649.7411650.339
B321649.6631648.2661648.356

Norm-matched and capped step magnitudes differ by 0.002%–0.036% across the four replay populations, calculated before rounding. This verifies close scalar step magnitudes under shared AdamW moments; update directions remain a separate quantity.

StateResponsesFixed CGIFixed relative gradient displacementCap active?
A80.6370.232Yes
A320.6910.404Yes
B80.4980.568Yes
B320.6140.537Yes

All four fixed CGI values exceed 0.30, so the active flags agree with the one-sided rule. Absolute 8-to-32 changes in fixed CGI are 0.054/0.116 for A/B; corresponding absolute gradient-displacement changes are 0.172/0.031. These quantify sensitivity to observer composition.

A shared-history replay and separately evolving training histories are different comparisons. AdamW’s moments and coordinatewise scaling remain part of the update interpretation. Parts 4–6 separately examine observer size over multiple controlled states.

Comments on the paper

7 comments
AuthorsOpenReview ↗

Official Response to Reviewer UEq7 (Part 1/5)

We thank the reviewer for the careful reading and constructive feedback. We address the questions and clarify the issues below.

About Baseline-Aware Comparison and Gradient Transport

What follows when two objectives have equal coefficient impact but different reference policies?

Experimental setting.

Two training roots (4104/4105), 20 snapshots/root at outer steps 0,6,…,1140,6,\ldots,114, eight hash-selected observer responses/snapshot, and the response-mean/token-mean reduction. Fixed controls: ratio-hard 0.013472934190594928; ratio-smooth 0.0828465938568115; TV-hard 0.004293510210083452; TV-smooth 0.027021306037902832. Spearman correlations are computed separately within each root/family/control across calibration and held-out snapshots, with the training root as the independent unit.

The qualification is explicit in §4.5. For positive baseline coefficient energy, ratio geometry uses the training anchor πk\pi_k, while behavior-anchored Binary-TV uses μ\mu. Equal CGI means equal fractional displacement from each declared baseline:

Ic,m=∥cm−cbase,m∥2∥cbase,m∥2.I_{c,m}=\frac{\lVert c_m-c_{{\rm base},m}\rVert_2}{\lVert c_{{\rm base},m}\rVert_2}.

Absolute displacement additionally incorporates baseline magnitude: ∥δcm∥2=Ic,m∥cbase,m∥2\lVert\delta c_m\rVert_2=I_{c,m}\lVert c_{{\rm base},m}\rVert_2.

A common-objective comparison also retains the baseline shift:

cm−c0=(cm−cbase,m)+(cbase,m−c0).c_m-c_0=(c_m-c_{{\rm base},m})+(c_{{\rm base},m}-c_0).

Declaring reference, reduction and population makes the coordinate interpretable without treating different objectives as interchangeable.

Equation 8 retains g=Zcg=Zc and δg=Zδc\delta g=Z\delta c. Token directions can reinforce or cancel, and AdamW further transforms the gradient. The smooth ceiling acts on the stated coefficient displacement; score-vector geometry and optimizer state determine its parameter-space effect.

At fixed controls, CGI–gradient correlations are 0.747/0.711 for TV-smooth and 0.657/0.644 for ratio-smooth, listing root 4104/root 4105. These within-root measurements show empirical gradient correspondence under the declared baselines.

Analytical example. With baseline coefficients (1,1)(1,1) and score map Z=(1,1)Z=(1,1), perturbations (a,a)(a,a) and (a,−a)(a,-a) have the same CGI ∣a∣|a|, but gradient changes 2a2a and zero. This is the directional distinction retained by Eq. 8. The Binary-TV bound ∣w−1∣≤1/δ|w-1|\leq1/\delta (§4.4) is a separate geometry-specific fact.

AuthorsOpenReview ↗

Official Response to Reviewer UEq7 (Part 2/5)

About Optimization Controls

How does tokenwise control differ from gradient clipping or learning-rate adjustment under matched settings?

Experimental setting.

Two retained final model/AdamW states (roots 4104/4105, next_step=120), immutable step 110 traces, nested hash-ranked observers of eight/32 responses from 256, response mean of token means, and behavior reference μ\mu. DRPO fixed/loose δ=0.027021306037902832\delta=0.027021306037902832, cap 0.30, clipping 1.0; stored BF16 parameters/moments; AdamW lr 10−610^{-6}, weight decay 0.01, betas (0.9,0.999), epsilon 10−810^{-8}. Each arm forks the same serialized state within its comparison.

Evidence from the submission (Table 9). The paper reports 0/5 observed collapses for the cap, compared with 3/5 for frozen fixed and gradient-norm matching, and 1/5 for BandPO. The cap’s AIME24 avg@32 point estimate is 31.2, compared with 28.7/29.1/30.9 for those controls. These results concern the original checkpoint cohort; the additional matched fixed/cap experiment is a separate cohort.

A global gradient clip rescales the assembled gradient, whereas a learning rate scales an optimizer-preconditioned step. Tokenwise coefficient reweighting acts before summation and can change the gradient direction. A single scalar generally cannot reproduce that transformation.

Under shared model and AdamW states, norm-matched and capped step magnitudes differ by 0.002%–0.036%. Thus the replay compares closely matched scalar step sizes while allowing token reweighting to change update direction.

The additional equal-calibration training experiment records fewer first-onset events for the cap in both smooth families at 120 and 240 steps. The full comparison, including reward and actual step norms, is reported in the response to XNY9, Part 7.

The experiments isolate tokenwise control under shared learning-rate, clipping and optimizer settings. Together, the training controls and AdamW replay examine coefficient reweighting and actual step magnitude under a common protocol.

AuthorsOpenReview ↗

Official Response to Reviewer UEq7 (Part 3/5)

About Ceiling Selection and Finite-Horizon Scope

How can a coefficient ceiling be used on a new model without training every candidate threshold?

Evidence already in the submission (Table 13). The paper applies the frozen 1.5B/BF16 source mapping without retuning to 1.5B/FP16, 4B/BF16 and 4B/FP16 rollout settings, retaining mean four-family rank correlations of 0.96/0.84/0.76. These are seed-level four-family rank correlations averaged over five runs. This existing transfer evidence supports reuse of the source mapping under the tested precision and scale changes.

Additional evidence already in the submission (Table 13). Across all three target settings using the unchanged source mapping, both smooth families have lower aggregate mean absolute relative target error than their hard counterparts. Together with the seed-level rank agreement, this provides complementary evidence on preserving family ordering and closeness to the declared coefficient-impact target under the tested precision and model-scale changes.

Choosing a coefficient budget and mapping it to a nominal threshold are separate decisions. For positive baseline energy, the smooth statistic is S=RMSu(X)S=\mathrm{RMS}_u(X). Equation 6 gives θ=S/I\theta=S/I for S,I>0S,I>0. When S=0S=0, the impact is zero and Eq. 10 retains the loose threshold. This avoids training each nominal threshold merely to map a chosen impact budget.

For a fixed scored population and declared smooth family, the entire control-to-impact curve follows from the single statistic SS. Candidate coefficient budgets can therefore be mapped through scalar calculations.

Equation 10 selects the smallest one-sided threshold adjustment that meets the local coefficient ceiling. Increasing θ\theta moves ww toward one. The budget has a precise coefficient-energy interpretation: for hard removal, CGI 0.30 corresponds to removing 9% of baseline coefficient energy. For the smooth families, the nominal threshold is obtained from the scored population through Eq. 6.

240-outer-step endpoint; the same five paired roots as at 120 steps.

FamilyFixed reward AUCCap reward AUCFixed eventsCap events
SPO0.4220.3675/52/5
DRPO0.6370.5445/52/5

At 240 steps, first-onset counts are 5/5 for fixed and 2/5 for capped control in both families, alongside the reward values shown above. The 120- and 240-step endpoints follow the same five paired roots.

The coefficient identities apply to objectives with the stated coefficient structure. The empirical evaluation covers the reported Qwen-based models and mathematical reasoning tasks, with the paired training comparison extended to 240 steps.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.