acceptodds
Under review as a conference paper at ICLR 2027

LNPO-ATC: LENGTH-NORMALIZED POLICY OPTIMIZATION WITH ADAPTIVE TOKEN CREDIT FOR LONG-CONTEXT REASONING

Abstract

Reinforcement learning has improved reasoning in large language models, but long reasoning trajectories remain challenging under finite generation budgets. Uniformly applying a terminal length penalty can turn positive content advantages into negative update coefficients throughout a response, coupling budget control with penalties on earlier reasoning tokens. We propose Length-Normalized Policy Optimization with Adaptive Token Credit (LNPO-ATC) to address this conflict. The method combines length-normalized sequence weighting, a smooth boundary penalty, and token-level allocation based on position relative to the available budget, while retaining the content advantage at every token. Our analysis separates penalty-strength and positional contributions and derives sufficient conditions for local improvement through alignment with the true content gradient. The evaluation covers six reasoning benchmarks, long-input generalization, and allocation controls. LNPO-ATC improves the six-task macro-average score on Qwen3.6-35B-A3B by an absolute 5.53% over GSPO; ATC adds an absolute 2.27% over uniform LNPO. The ordered-versus-shuffled comparison, which also varies EOS weighting, yields an absolute 1.79% gain. At equal counts of 2,000 updates, training wall time is 1.4–1.5% higher than GSPO on the two Qwen3.6 backbones.

Primary area:
reinforcement learning
Keywords:
Stochastic Optimization, Policy Gradient Estimation, Sampling Design, Variance Reduction, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JvJGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

this paper: how to optimize the long-seq inference under limited generation budget.

original way: length penalty equally boardcast to each traj, making tokens with pos content adv. get neg update, which is unreasonable.

core: propose LNPO-ATC, adding length-normalized sequence weighting、smooth boundary penalty and adaptive token-level credit allocation together. using a cubic soft barrier to assign length penalty based on token vs. generation budget.

exps on mutiple benchmark shows the improvements, under some specific condition.

Critical strengths
  1. motivation is good, method make sense.

  2. exp results seems sufficient and consistent.

  3. ablations seems fine.

Critical weaknesses
  1. therotical assumptions may not valid in LLM training.

  2. More analysis need to be done for affects on contributions on EOS token.

  3. need more anaysis on generation budget and hyperparams like /alpha, (see table 6. on \alpha = 0.8 and 0.9)

Questions for authors

see above

3 comments

1. Theory and practical training.

LNPO-ATC guides training through budget-relative token credit allocation, preserving early correctness-learning signals while progressively increasing length penalties near the output limit to balance reasoning accuracy and generation efficiency.

Algorithm 1 samples responses from a frozen policy and normalizes verifier rewards within each group into Aᶜ. Each valid token receives Aᶜ−βbα(t;H). The length term is zero before αH and rises smoothly toward the cap. Detached coefficients weight the clipped GSPO-token objective; backpropagation and AdamW update the parameters. The rollout policy is refreshed before the next batch. Defaults are 16 responses/prompt, α=0.8, β=0.9, cap 32,768 and 2,000 updates.

Submitted Tables 1/2 report Qwen3.6-35B-A3B accuracy (%, mean ± SD) and mean output length:

MethodHMMT February 2026IMO-AnswerBenchLiveCodeBench v6LCB ProZebraLogicGPQA DiamondLength (k)
GSPO85.4 ± 0.7682.3 ± 0.8383.4 ± 0.5946.5 ± 1.0368.9 ± 0.9486.8 ± 0.439.59
LNPO-ATC90.3 ± 0.5887.1 ± 0.6588.4 ± 0.4253.5 ± 0.8677.2 ± 0.7490.0 ± 0.317.47

LNPO-ATC has higher mean accuracy than GSPO on all six datasets, with gains of 3.2–8.3 percentage points, while mean output length falls by 22.1%. These results support the effectiveness of integrating our algorithm (LNPO-ATC) into policy training in the reported setting.

2. EOS contribution.

EOS-fixed shuffle fixes the sampled-EOS coefficient and permutes valid non-EOS coefficients, preserving their multiset; both methods use the Q1 settings, cubic allocation and five runs per condition.

Accuracy (%, mean ± SD).

MethodHMMT February 2026IMO-AnswerBenchLiveCodeBench v6LCB ProZebraLogicGPQA Diamond
EOS-fixed shuffle89.7 ± 1.6686.0 ± 0.4787.5 ± 0.3552.0 ± 0.3375.7 ± 0.3490.0 ± 0.42
LNPO-ATC (full)90.3 ± 0.5887.1 ± 0.6588.4 ± 0.4253.5 ± 0.8677.2 ± 0.7490.0 ± 0.31

With the EOS coefficient rule fixed, LNPO-ATC matches or exceeds the shuffled control in mean accuracy on all six datasets, supporting non-EOS placement.

3. α and generation budgets.

The α/β settings and LCB v6 means match Table 6. GSPO/default scores use Table 1; hard-cut HMMT/LCB v6 use Table 5; remaining cells are independent E2 summaries.

Parameter sensitivity (accuracy %, mean ± SD).

αβHMMT February 2026IMO-AnswerBenchLiveCodeBench v6LCB ProZebraLogicGPQA Diamond
0.50.989.1 ± 1.785.7 ± 0.486.1 ± 0.552.2 ± 0.375.6 ± 0.589.6 ± 0.5
0.8085.4 ± 0.882.3 ± 0.883.4 ± 0.646.5 ± 1.068.9 ± 0.986.8 ± 0.4
0.80.589.7 ± 1.786.3 ± 0.387.5 ± 0.352.7 ± 0.476.5 ± 0.489.9 ± 0.4
0.80.890.3 ± 1.486.9 ± 0.588.2 ± 0.353.4 ± 0.477.2 ± 0.389.9 ± 0.4
0.80.990.3 ± 0.687.1 ± 0.788.4 ± 0.453.5 ± 0.977.2 ± 0.790.0 ± 0.3
0.80.9590.3 ± 1.487.0 ± 0.588.2 ± 0.453.3 ± 0.376.9 ± 0.690.1 ± 0.6
0.80.9990.3 ± 1.486.7 ± 0.487.9 ± 0.353.1 ± 0.577.0 ± 0.590.1 ± 0.3
0.8289.7 ± 1.784.8 ± 0.285.6 ± 0.451.7 ± 0.375.8 ± 0.389.7 ± 0.3
0.90.989.7 ± 1.786.8 ± 0.687.7 ± 0.653.3 ± 0.476.7 ± 0.590.2 ± 0.3
1 (hard)0.987.3 ± 0.481.6 ± 0.284.6 ± 0.447.5 ± 0.473.4 ± 0.290.5 ± 0.4

At β=0.9, α=0.8→0.9 lowers manuscript LCB v6 accuracy 88.4%→87.7% (−0.7 pp) and the six-task mean 81.08%→80.76% (−0.32 pp). At H=32768, the penalty starts at tokens 26,215/29,492: the latter halves the warning width and doubles the maximum cubic slope. The α=1 row uses the separate hard-cut rule.

Generation budgets (accuracy %, mean ± SD).

We vary inference caps; the training cap is 32,768 tokens.

Output capMethodHMMT February 2026IMO-AnswerBenchLiveCodeBench v6LCB ProZebraLogicGPQA Diamond
8192GSPO80.61 ± 1.6678.35 ± 0.4277.33 ± 0.4240.27 ± 0.4863.90 ± 0.5884.85 ± 0.36
8192LNPO-ATC87.27 ± 1.3683.25 ± 0.4084.42 ± 0.4248.29 ± 0.1773.40 ± 0.3188.59 ± 0.45
16384GSPO85.45 ± 1.3681.90 ± 0.3883.24 ± 0.2845.65 ± 0.3968.32 ± 0.4386.16 ± 0.58
16384LNPO-ATC90.30 ± 1.3687.15 ± 0.4288.55 ± 0.5253.63 ± 0.3177.26 ± 0.3390.10 ± 0.28
65536GSPO86.67 ± 2.7185.40 ± 0.4586.81 ± 0.5650.65 ± 0.3772.82 ± 0.2989.70 ± 0.58
65536LNPO-ATC90.30 ± 1.3687.50 ± 0.0089.04 ± 0.2254.35 ± 0.5177.82 ± 0.2990.40 ± 0.36

LNPO-ATC has higher mean accuracy than GSPO in all 18 budget–dataset pairs. Equal-weight mean gains narrow to 6.65/6.04/2.90 pp at 8,192/16,384/65,536 tokens; per-task gains at 65,536 are 0.71–5.00 pp.

Reviewer_JvJGOpenReview ↗

Thank you

Thank you for your rebuttal, it partially solves my concerns. Therefore, I keep my score. (I slightly improve scores, but I dont think it reach to acceptance score. i.e., from 2.6(initial)->3.3(now), so it is still 3).

AuthorsOpenReview ↗

Thank you for your feedback

Thank you for taking the time to read our rebuttal and reconsider our work. We appreciate your feedback!

Reviewer_5SQnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Length-Normalized Policy Optimization with Adaptive Token Credit (LNPO-ATC) for long-context reasoning. The method combines length-normalized sequence weighting, a smooth boundary penalty, and budget-relative token-level allocation to avoid penalizing earlier reasoning tokens. The authors further provide theoretical analysis of penalty strength and positional contributions, along with sufficient conditions for local policy improvement. Experiments on six reasoning benchmarks demonstrate improved reasoning performance and reduced generation length.

Critical strengths
  1. The paper addresses an important problem in long-context reasoning by examining the limitations of uniform length penalties. The proposed LNPO-ATC uses budget-relative token-level allocation to retain the content advantage at every token while avoiding explicit length penalties before the warning region.
  2. The paper provides theoretical analysis that separates penalty-strength and positional contributions, along with sufficient conditions for local content improvement. Experiments on six reasoning benchmarks and two Qwen3.6 backbones demonstrate improvements in reasoning performance. The allocation controls further distinguish the effects of penalty strength, nonuniform weighting, and temporal placement.
Critical weaknesses
  1. The novelty of LNPO-ATC is relatively limited. The method builds on the existing GSPO and GSPO-token constructions, primarily introducing a smooth boundary penalty and budget-relative token-level allocation. Although the theoretical analysis provides useful insights, the core algorithmic design is relatively straightforward, and its differences from existing length control methods could be further clarified.
  2. The experimental evaluation does not fully establish the generalizability and independent contribution of ATC. Most experiments use a fixed 32,768-token generation budget, leaving its effectiveness under different budgets unclear. In addition, the ordered-versus-shuffled comparison in Table 4 also varies EOS weighting, so the contribution of temporal placement cannot be fully isolated.
Questions for authors
  1. Could the authors further clarify the differences between LNPO-ATC and existing length control methods, particularly regarding the contribution of budget-relative token-level allocation?
  2. How does LNPO-ATC perform under different generation budgets, such as 8k, 16k, and 64k tokens? Are the improvements consistent across different budget settings?
  3. In Table 4, the ordered-versus-shuffled comparison also varies EOS weighting. Could the authors provide an additional comparison with a fixed EOS coefficient to better isolate the contribution of temporal placement?
  4. Could the authors report the actual training token consumption and compare LNPO-ATC with the baselines under matched computational budgets to further clarify its training efficiency?
3 comments

Q1. Contribution.

LNPO-ATC controls where length pressure acts during training. Uniform LNPO broadcasts the terminal penalty to every token, potentially suppressing useful early reasoning. LNPO-ATC retains correctness credit throughout and applies the length penalty only from αH to H (H: cap; α: onset; β: strength), preserving early learning while controlling late generation.

We retain GSPO's sequence weighting and token-gradient construction. LCPO targets requested lengths, LAPO adapts reasoning length, and DAPO applies soft overlong punishment; our contribution is budget-relative allocation of length pressure within each response.

Table 4 tests allocation at matched response-level penalty strength. Here c is the terminal cost and q̄ the original ATC mean (common β omitted); Matched ATC rescales ATC to mean c, and Uniform at q̄ uses q̄ at every token. Scores average six tasks.

AllocationMeanScoreContrast (pp) [95% CI]
Uniform LNPOc78.82±0.34Reference
Matched shufflec78.87±0.28Matched ATC − Matched shuffle: +2.00 [1.40, 2.59]
Matched ATCc80.86±0.25Matched ATC − Uniform LNPO: +2.05 [1.69, 2.40]
Uniform at q̄q̄79.24±0.22Uniform at q̄ − Uniform LNPO: +0.42 [0.04, 0.80]
Shuffled ATCq̄79.30±0.27LNPO-ATC − Shuffled ATC: +1.79 [1.41, 2.16]
LNPO-ATCq̄81.08±0.19LNPO-ATC − Uniform at q̄: +1.85 [1.60, 2.10]

Matched-strength comparisons support allocation gains beyond mean-strength reduction (Proposition 1). This motivates testing LNPO-ATC in long-horizon agent RL.

Q2. Generation budgets.

The comparison fixes each method's U2000 checkpoint from training with a 32,768-token cap and varies only the inference output cap, with tasks, templates, verification and decoding settings matched.

LNPO-ATC improves over GSPO in all 18 cap–dataset pairs; six-task mean gains narrow to 6.65/6.04/2.90 pp at 8,192/16,384/65,536 tokens. Scores summarize five scenario runs.

CapMethodHMMTIMOLCBv6LCBProZebraGPQA
8192GSPO80.61±1.6678.35±0.4277.33±0.4240.27±0.4863.90±0.5884.85±0.36
8192LNPO-ATC87.27±1.3683.25±0.4084.42±0.4248.29±0.1773.40±0.3188.59±0.45
16384GSPO85.45±1.3681.90±0.3883.24±0.2845.65±0.3968.32±0.4386.16±0.58
16384LNPO-ATC90.30±1.3687.15±0.4288.55±0.5253.63±0.3177.26±0.3390.10±0.28
65536GSPO86.67±2.7185.40±0.4586.81±0.5650.65±0.3772.82±0.2989.70±0.58
65536LNPO-ATC90.30±1.3687.50±0.0089.04±0.2254.35±0.5177.82±0.2990.40±0.36

Q3. Fixed-EOS control.

EOS-fixed shuffle preserves the sampled-EOS coefficient Aᶜ−βbα(e;H) and shuffles only valid non-EOS coefficients while preserving their multiset, mass and squared-weight sum. The reported setup uses Qwen3.6-35B-A3B, cubic allocation, α=0.8, β=0.9, a 32,768-token cap and 2,000 updates, with five runs per condition.

Accuracy (%, mean±SD).

MethodHMMTIMOLCBv6LCBProZebraGPQA
EOS-fixed shuffle89.6970±1.6686.0000±0.4787.4692±0.3551.9521±0.3375.7400±0.3490.0000±0.42
LNPO-ATC (full)90.3±0.5887.1±0.6588.4±0.4253.5±0.8677.2±0.7490.0±0.31

These comparisons support an additional contribution from ATC’s non-EOS positional allocation: retaining the original placement improves task accuracy under a fixed EOS coefficient rule.

Q4. Training efficiency.

Manuscript setup: Qwen3.6-35B-A3B, 2,000 updates. Generated training tokens (B=10^9; mean±sample SD, five runs): GSPO 41.085±1.249; LNPO-ATC 35.257±0.335.

The supplied four-node results compare GSPO and LNPO-ATC on Qwen3.6-35B-A3B, using the same initial base, 16 H100 GPUs per method, five runs, and a 32,768-token training/evaluation cap. Each checkpoint is selected by cumulative resource consumption as the latest saved checkpoint at or below a prespecified limit; saves occur every ten updates, and the largest shortfall is 0.9551%. The 1,500 GPU-hour point extends training beyond U2000 with optimizer state and the fixed 1e-6 learning rate retained.

GPU-hoursMethodHMMTIMOLCBv6LCBProZebraGPQA
600GSPO83.64±1.6679.55±1.4280.74±0.7842.74±1.4365.16±1.7683.84±0.94
600LNPO-ATC87.88±2.1484.65±0.6585.97±0.5649.79±1.1673.92±0.8387.88±0.94
900GSPO84.85±0.0081.05±1.1482.29±1.0944.97±1.0367.40±1.0285.56±0.77
900LNPO-ATC90.91±2.1487.00±0.8388.38±0.3253.77±1.1877.10±0.7989.80±0.42
1,200GSPO82.42±11.2281.80±2.7882.84±1.4346.20±2.6468.46±0.7185.96±1.80
1,200LNPO-ATC90.30±7.5586.95±1.1488.19±1.0452.98±2.2974.58±1.5588.38±2.40
1,500GSPO86.06±1.6682.80±1.0883.66±0.4247.16±1.5569.58±0.6686.97±0.66
1,500LNPO-ATC90.30±1.3687.15±0.6088.44±0.7353.60±1.1176.58±0.4790.00±0.97

LNPO-ATC has higher means in all 24 comparisons and uses 15.11%–15.55% fewer training tokens. At 900 GPU-hours it exceeds GSPO at 1,500 on all tasks (+2.83–7.52 pp); at 1,500 its gains remain +3.03–7.00 pp.

Thank you for the detailed response and clarifications. Most of my concerns have been adequately addressed. I will maintain my original rating.

Thank you for taking the time to read our response. We’re glad we were able to address most of your concerns, and we appreciate your helpful feedback.

Reviewer_dJxAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a position-dependent allocation of the terminal length penalty in budget-aware GSPO where early tokens keep the pure content advantage while late tokens and the EOS carry the penalty.

Critical strengths

The experimental design is extensive: six allocation rules are compared with paired intervals, the backbones (35B-A3B, 27B) are at a scale that matters in practice, and six hard benchmarks with five runs are appropriate for long-trajectory behavior.

Critical weaknesses
  1. The placement effect is not isolated from the EOS/terminal weighting.
  2. The length reduction is sensitive to the penalty strength, while the cost and evaluation protocol are not fully reported.
Questions for authors
  1. W1: The ordered-vs-shuffled contrast in Table 4 changes the positions that carry the penalty and the coefficient on the sampled EOS. Please add a variant that fixes the EOS coefficient and only changes the non-EOS ones.
  2. W2: In Table 6, only two nonzero beta yields outputs shorter than GSPO, and the length is non-monotone in it. Please report run-level length and truncation rate with their SDs for every beta and alpha.
  3. W2: Please report generated training tokens, and what the wall-time includes.
1 comment

Thank you for the detailed review.

W1. EOS-fixed shuffle.

To isolate the contribution of non-EOS positional allocation, we compare full LNPO-ATC with EOS-fixed shuffle, which fixes the sampled EOS coefficient Aᶜ−βbα(e;H) and shuffles only valid non-EOS coefficients while preserving their multiset, sum and sum of squares. Both conditions use Qwen3.6-35B-A3B, cubic allocation, α=0.8, β=0.9, a 32,768-token cap, 2,000 updates and five runs; full-method accuracy quotes manuscript Table 1; length/truncation statistics and shuffle results use E1.

EOS comparison: dataset accuracy (%, mean ± SD), length and truncation.

MethodHMMT February 2026IMO-AnswerBenchLiveCodeBench v6LCB ProZebraLogicGPQA DiamondLength (k tokens)Truncation (%)
EOS-fixed shuffle89.70 ± 1.6686.00 ± 0.4787.47 ± 0.3551.95 ± 0.3375.74 ± 0.3490.00 ± 0.427.78 ± 0.1593.41 ± 0.18
LNPO-ATC (full)90.30 ± 0.5887.10 ± 0.6588.40 ± 0.4253.50 ± 0.8677.20 ± 0.7490.00 ± 0.317.47 ± 0.1172.97 ± 0.29

With the EOS coefficient rule fixed, full LNPO-ATC scores 0.60/1.10/0.93/1.55/1.46 pp higher on the first five datasets and ties on GPQA, with 0.31k tokens shorter outputs and 0.44 pp less truncation, supporting the contribution of non-EOS positional allocation.

W2a. Complete parameter statistics.

For each α and β configuration, we provide the means and standard deviations of output length and truncation rate across five runs.

The α/β settings and LCB v6 means match Table 6. GSPO/default scores use Table 1; hard-cut HMMT/LCB v6 use Table 5; remaining cells are independent E2 summaries.

Parameter sensitivity (accuracy in %, mean ± SD).

αβHMMT February 2026IMO-AnswerBenchLiveCodeBench v6LCB ProZebraLogicGPQA DiamondLength (k tokens)Truncation (%)
0.50.989.1 ± 1.785.7 ± 0.486.1 ± 0.552.2 ± 0.375.6 ± 0.589.6 ± 0.58.3 ± 0.24.2 ± 0.3
0.8085.4 ± 0.882.3 ± 0.883.4 ± 0.646.5 ± 1.068.9 ± 0.986.8 ± 0.49.6 ± 0.27.4 ± 0.3
0.80.589.7 ± 1.786.3 ± 0.387.5 ± 0.352.7 ± 0.476.5 ± 0.489.9 ± 0.412.1 ± 0.39.8 ± 0.6
0.80.890.3 ± 1.486.9 ± 0.588.2 ± 0.353.4 ± 0.477.2 ± 0.389.9 ± 0.411.5 ± 0.28.9 ± 0.5
0.80.990.3 ± 0.687.1 ± 0.788.4 ± 0.453.5 ± 0.977.2 ± 0.790.0 ± 0.37.5 ± 0.13.0 ± 0.3
0.80.9590.3 ± 1.487.0 ± 0.588.2 ± 0.453.3 ± 0.376.9 ± 0.690.1 ± 0.611.0 ± 0.28.5 ± 0.5
0.80.9990.3 ± 1.486.7 ± 0.487.9 ± 0.353.1 ± 0.577.0 ± 0.590.1 ± 0.310.8 ± 0.28.8 ± 0.6
0.8289.7 ± 1.784.8 ± 0.285.6 ± 0.451.7 ± 0.375.8 ± 0.389.7 ± 0.38.6 ± 0.23.9 ± 0.2
0.90.989.7 ± 1.786.8 ± 0.687.7 ± 0.653.3 ± 0.476.7 ± 0.590.2 ± 0.312.4 ± 0.310.3 ± 0.6
1 (hard)0.987.3 ± 0.481.6 ± 0.284.6 ± 0.447.5 ± 0.473.4 ± 0.290.5 ± 0.413.0 ± 0.311.7 ± 0.5

At α=0.8, every tested nonzero β has a higher mean score than the manuscript GSPO reference on all six datasets. For the tested β∈{0.5,0.8,0.9,0.95,0.99}, all six means also exceed the manuscript uniform-LNPO reference. The length response remains non-monotone: only β=0.9 and 2 shorten outputs versus GSPO. Raising α from 0.8 to 0.9 at β=0.9 increases manuscript mean length from 7.47k to 12.4k and supplementary truncation from 3.0% to 10.3%. The shared default counts once; β=0 is GSPO and α=1 is the separate hard-cut rule.

W2b. Generated training tokens and wall-time scope.

ATC reduces generated training tokens by 14.18% on MoE and 13.32% on dense; manuscript wall-times increase by 1.4% and 1.5%, respectively.

Both models use 2,000 updates and 16/32 H100 SXM 80GB GPUs (MoE/dense). Token counts are mean ± sample SD over five runs (B=10^9); training times quote Table 8.

ModelMethodGPUsGenerated training tokens (B), mean ± SD
35B-A3BGSPO1641.085 ± 1.249
35B-A3BATC1635.257 ± 0.335
27B denseGSPO3245.858 ± 1.134
27B denseATC3239.751 ± 0.792

Wall-time would run from after all GPUs are allocated, before initialization, until update 2,000 and all required validation, final checkpoint writes, asynchronous I/O and synchronization finish. It would include initialization and warm-up, rollouts, reward computation, model updates, in-training validation, checkpointing and waits, excluding pre-allocation queueing, preparation completed before timing starts, and independent post-training evaluation and analysis. Rollouts would use the same 16/32 GPUs reserved throughout, including CPU-reward waits, and overlapping work counts once, so GPU-hours = GPU count × wall-hours.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.