- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
this paper: how to optimize the long-seq inference under limited generation budget.
original way: length penalty equally boardcast to each traj, making tokens with pos content adv. get neg update, which is unreasonable.
core: propose LNPO-ATC, adding length-normalized sequence weighting、smooth boundary penalty and adaptive token-level credit allocation together. using a cubic soft barrier to assign length penalty based on token vs. generation budget.
exps on mutiple benchmark shows the improvements, under some specific condition.
Critical strengths
-
motivation is good, method make sense.
-
exp results seems sufficient and consistent.
-
ablations seems fine.
Critical weaknesses
-
therotical assumptions may not valid in LLM training.
-
More analysis need to be done for affects on contributions on EOS token.
-
need more anaysis on generation budget and hyperparams like /alpha, (see table 6. on \alpha = 0.8 and 0.9)
Questions for authors
see above
3 comments
Thank you
Thank you for your rebuttal, it partially solves my concerns. Therefore, I keep my score. (I slightly improve scores, but I dont think it reach to acceptance score. i.e., from 2.6(initial)->3.3(now), so it is still 3).
Thank you for your feedback
Thank you for taking the time to read our rebuttal and reconsider our work. We appreciate your feedback!
1. Theory and practical training.
LNPO-ATC guides training through budget-relative token credit allocation, preserving early correctness-learning signals while progressively increasing length penalties near the output limit to balance reasoning accuracy and generation efficiency.
Algorithm 1 samples responses from a frozen policy and normalizes verifier rewards within each group into Aᶜ. Each valid token receives Aᶜ−βbα(t;H). The length term is zero before αH and rises smoothly toward the cap. Detached coefficients weight the clipped GSPO-token objective; backpropagation and AdamW update the parameters. The rollout policy is refreshed before the next batch. Defaults are 16 responses/prompt, α=0.8, β=0.9, cap 32,768 and 2,000 updates.
Submitted Tables 1/2 report Qwen3.6-35B-A3B accuracy (%, mean ± SD) and mean output length:
LNPO-ATC has higher mean accuracy than GSPO on all six datasets, with gains of 3.2–8.3 percentage points, while mean output length falls by 22.1%. These results support the effectiveness of integrating our algorithm (LNPO-ATC) into policy training in the reported setting.
2. EOS contribution.
EOS-fixed shuffle fixes the sampled-EOS coefficient and permutes valid non-EOS coefficients, preserving their multiset; both methods use the Q1 settings, cubic allocation and five runs per condition.
Accuracy (%, mean ± SD).
With the EOS coefficient rule fixed, LNPO-ATC matches or exceeds the shuffled control in mean accuracy on all six datasets, supporting non-EOS placement.
3. α and generation budgets.
The α/β settings and LCB v6 means match Table 6. GSPO/default scores use Table 1; hard-cut HMMT/LCB v6 use Table 5; remaining cells are independent E2 summaries.
Parameter sensitivity (accuracy %, mean ± SD).
At β=0.9, α=0.8→0.9 lowers manuscript LCB v6 accuracy 88.4%→87.7% (−0.7 pp) and the six-task mean 81.08%→80.76% (−0.32 pp). At H=32768, the penalty starts at tokens 26,215/29,492: the latter halves the warning width and doubles the maximum cubic slope. The α=1 row uses the separate hard-cut rule.
Generation budgets (accuracy %, mean ± SD).
We vary inference caps; the training cap is 32,768 tokens.
LNPO-ATC has higher mean accuracy than GSPO in all 18 budget–dataset pairs. Equal-weight mean gains narrow to 6.65/6.04/2.90 pp at 8,192/16,384/65,536 tokens; per-task gains at 65,536 are 0.71–5.00 pp.