- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies why response length grows during reinforcement learning for competitive programming, and how to limit it without losing the long reasoning that hard problems need. The analysis finds that the largest proportional growth occurs on problems the initial model already solves reliably, while accuracy gains concentrate on hard problems, and that negative updates on failed rollouts drive much of the early growth. Penalizing truncated responses instead of masking them shortens outputs but loses long successes, and enlarging the fixed budget does not help. The proposed recipe caps each prompt at a multiple of the initial model's median successful length, keeps the full budget for rarely solved prompts, and masks responses stopped at the cap. It cuts response length substantially at comparable accuracy and transfers to a second model family.
Critical strengths
-
The analysis would carry a submission on its own. Decomposing length growth by initial solvability rather than by difficulty labels yields a concrete finding, namely that growth concentrates where brief answers already succeeded. The sign-restricted runs are a clean intervention, and the decomposition by completion cutoff in Figure 4b correctly localizes the cost of penalizing truncation to successes beyond the training window rather than to shortening itself. The paper also reports that a 256k fixed budget does not beat the 102k baseline, a negative result many authors would have dropped.
-
The length reductions are large, and the baselines are trained under matched conditions rather than taken from released checkpoints. Recomputing them, the reduction against the baseline is 31.8 percent on LiveCodeBench and 24.5 percent on LiveCodeBench-Pro, and 6 to 17 percent against the two length penalties, all far outside the reported spread. The multiplier can also be calibrated before training from the share of initial rollouts its caps would stop, which I checked is internally consistent with the reported lengths.
Critical weaknesses
-
The accuracy side of the Pareto claim is not separated from noise. There is one training run per configuration, and the error bars cover decoding seeds only. The accuracy advantage over the linear length penalty on LiveCodeBench is 0.22 points against a combined standard error of 0.57, and over the baseline 0.33 against 0.51. On LiveCodeBench-Pro the advantages over the two penalties are 0.87 and 1.09 points against standard errors near 0.6 to 0.7. None reaches conventional separation, and none includes training-run variability, which for RL post-training is usually the larger component. I would keep the length claims, which are solid, and weaken the accuracy claims to comparable rather than better. Selecting each checkpoint on LiveCodeBench accuracy and then reporting that metric compounds this, and the resulting bias need not be equal across methods.
-
The natural control is missing. Adaptive budgets change three things relative to the baseline, namely per-prompt caps, a 256k ceiling instead of 102k, and masking at that cap. Section 4.3 reports that a 256k run with overlong filtering was trained but omits it, yet that is exactly the configuration differing from the proposed method only in the per-prompt caps. Its numbers are needed to attribute the effect.
Questions for authors
-
Can you report the omitted 256k with overlong filtering run in Table 1, with accuracy and length on both coding benchmarks?
-
How much do accuracies move across training runs? Two or three repeated runs for the baseline and the proposed method would show whether the accuracy ordering is stable.
-
How would Table 1 change if checkpoints were selected on a held-out split rather than on LiveCodeBench accuracy?
-
What is the cost of the reference pass relative to one training run, in the same units as the step times in Section 6.6? With 16 samples per prompt over 3,532 prompts at long context it is not negligible.
Thank you for the thoughtful review. Please see our response to each of your points below.
We agree that on LiveCodeBench (LCB) the accuracy gain of our method over the linear length penalty is within the standard error, and we will revise the paper to describe LCB accuracy as comparable rather than better. Our central claim is a better accuracy–length tradeoff: shorter responses at comparable accuracy.
On LiveCodeBench-Pro (LCBPro), the advantage over the two length penalties is small, but it is larger than the error bars: about 1.5 times the combined standard error. Adaptive budgets also use fewer tokens than these two methods, so against them our method is both slightly more accurate and shorter. We will make this claim more precise in the paper.
On training variance, due to resource constraints we are not able to repeat every configuration, so we trained two more baseline runs with different seeds and evaluated them as in Table 1.
The baseline is relatively stable across runs. The standard deviation of accuracy is no larger than the baseline's decoding standard error in Table 1 (0.3 and 0.5 points). The standard deviation of length is about 1%-2%, far below the reductions from adaptive budgets.
Thank you for raising this control. At a 256k window, truncation is very rare. In the 256k run of Table 1, no rollout was truncated in the first half of training, and at most 1% of a step's rollouts were truncated up to the selected checkpoint. Overlong filtering changes only how truncated responses are treated, so it has almost nothing to act on here, which is why we did not include the run with filtering.
To address this concern, we report the 256k run with overlong filtering below. Its results differ little from the run without filtering.
This indicates that the shorter responses of adaptive budgets come from the per-prompt caps, not from the larger window or from masking at it. We will add this run in the revision.
Checkpoints were selected on LCB only. LCBPro were not used for selection, so it's numbers in Table 1 do not depend on selecting on the reported metric. To check the LCB numbers in the same way, we reselected each run's checkpoint by LCBPro accuracy.
Selecting on LCBPro does not change the ranking of the methods, and adaptive budgets still give the best accuracy–length tradeoff.
The cost depends on the RL infra, because the reference pass is generation only. It needs no trainer and can run fully in parallel on inference nodes. In our setup, the reference pass costs about 15% of the wallclock time of one RL training run under same resource.
As Section 5 notes, this kind of profiling is usual in RL training, where it saves compute: prompts that are too easy or too hard for the initial model give no training signal. Common pipelines therefore sample the initial model on the training prompts and filter them by pass rate [1, 2], and the solve rates and successful lengths can be taken from that same pass.
[1] Kimi Team. Kimi k1.5: Scaling Reinforcement Learning with LLMs.
[2] He et al. Skywork Open Reasoner 1 Technical Report.