- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper studies how to expose information about an upcoming response’s length before generation while preserving the target language model’s original output distribution. The authors propose Length-Separated Sampling, which combines an advance random choice of short or long regime, length-predictor guidance, per-token probability constraints, and regime-posterior updates to construct a marginal-preserving conditional sampling process. Experiments report length separation across 11 models and three datasets. The research question and proposed approach have merit, but the evaluation primarily relies on older datasets and small to medium-sized models, and the writing has clear shortcomings. Both the empirical evaluation and the presentation require further refinement.
Critical strengths
The paper poses an interesting and practically relevant research question: can information about response length be exposed before generation while preserving the language model’s original output distribution? Stochastic responses to the same prompt can differ substantially in length, leaving uncertainty that prompt-level length prediction alone cannot resolve. This question is worth investigating.
Critical weaknesses
-
The experimental coverage and representativeness of the length distributions are insufficient to establish broad value for current reasoning and serving scenarios. Effectiveness experiments primarily rely on older, small to medium-sized models, and tasks are limited. The most convincing gains attributable to the learned predictor are concentrated on relatively simple GSM8K problems, without validation on complex, extended reasoning.
-
In several writing and instruction settings, the fixed termination rule matches or exceeds the full method. The paper primarily reports ratios of regime mean lengths without quantifying how much the regime reduces prediction error or uncertainty beyond knowing the prompt, or demonstrating corresponding scheduling, latency, or resource benefits. It therefore remains unclear how much practical value the learned predictor provides relative to its additional complexity and computational cost.
-
The writing and technical exposition need substantial refinement. For example, several citations use narrative formatting without appropriate parentheses when serving as sentence-final references.
Questions for authors
-
Does the learned predictor retain substantial benefits over the termination-only rule on a more challenging reasoning task and a more recent, stronger open model? What primarily limits performance in settings such as Open-RS2, where the oracle indicates considerable available separation but the method achieves little?
-
What is the maximum generation length, and what fraction of responses reaches this cap, particularly for Qwen3-8B? If a substantial fraction is truncated, the cap could suppress within-prompt length variation and partly explain the reported Qwen3-8B exception. Do the low variance and weak separation persist under a larger generation budget?
7 comments
Response to Reviewer Pb2J (2/2): generation caps, harder tasks, Open-RS2, writing
3. "If a substantial fraction is truncated, the cap could suppress within-prompt length variation and partly explain the reported Qwen3-8B exception" (Q2)
We tested this and it is not the explanation. Generation caps are 2,048–6,144 tokens depending on the model (the revision will state them, with truncation rates). Only 1.3–1.4% of Qwen3-8B's GSM8K responses reach the cap. We then raised its cap to 16,384, so that no response is truncated (new). Within-prompt variation is 17% of its total length variance, unchanged from 18% at the lower cap, against 47–48% for AceReason-7B and Open-RS3 at the same cap. The best achievable split also stays low (1.45, against 3.1–3.4 for those two models). Qwen3-8B simply varies little from sample to sample, so there is little for any method to separate.
4. "older, small to medium-sized models ... without validation on complex, extended reasoning" (W1); "a more challenging reasoning task" (Q1)
Models. The method needs a model's logits and hidden states, so it can only be evaluated on open-weight models. Within that constraint the evaluation is broad: eleven models in the paper, ten of them released in 2025, covering RL-trained reasoning models, distilled reasoning models and non-thinking models. With the three models added here it now spans 1.5B to 70B parameters.
Tasks. GSM8K problems are simple, but the responses are not short: these models produce 600–1,300 tokens of reasoning per problem, and it is the length of this reasoning that the method separates. To test a harder task we ran AceReason-7B on MATH Levels 1–2 (new), where its responses average 3,000 tokens. The method separates them by 1.14× [95% CI 1.12, 1.16], about 400 tokens per response. On MATH Level 5 our current predictor does not separate the regimes (1.02 and 1.05 on two models). The revision will state this as a limitation.
5. "What primarily limits performance in settings such as Open-RS2, where the oracle indicates considerable available separation but the method achieves little?" (Q1)
The limit is how our standard predictor was calibrated for this one model, not a lack of length signal. For Open-RS2 on GSM8K, the tilt built from the standard predictor pushed the regimes in the wrong direction on validation data. Our procedure detected this and fell back to using no predictor, so the Table 1 entry (1.09) is the no-predictor version.
The signal is there. We trained a variant of the predictor that outputs the probability of finishing early directly, so there is no tilt strength to calibrate. On the same model and dataset it reaches 1.96 on the Table 1 statistic, against an oracle of 2.49. Open-RS2 is therefore in line with the other reasoning models once the calibration problem is removed. The revision will report this variant.
6. "The writing and technical exposition need substantial refinement" (W3)
The citation formatting is fixed throughout. Section 4 will present a single construction, with the accept/reject alternative moved to the appendix (also requested by Reviewer 92Cw). We would welcome pointers to any other passages.
We hope these answers and the new experiments address the reviewer's concerns. If any remain, we would be grateful if the reviewer could say which, so that we can address them during the discussion period.
Thank you for the detailed response and the additional experiments on larger models. I also appreciate the improvements to the paper’s clarity and presentation. These results help address my concern about model coverage. However, I still see a substantial gap between the current evaluation and practical deployment in challenging reasoning settings. The positive results on MATH are limited to Levels 1–2, while the reported lack of separation on Level 5 raises concerns about effectiveness on more demanding problems, such as AIME-level questions.
I have increased my score to reflect the additional evidence and improved presentation. However, while I appreciate the acknowledgment of this limitation, I would need stronger validation on challenging reasoning workloads representative of practical use to support a higher score.
Response to Reviewer Pb2J (1/2): termination rule, larger models, practical benefit
We thank the reviewer for the review. We answer each point in turn. Experiments marked (new) were run for this response; we are adding them to the paper and will upload the revised PDF during the discussion period.
1. "In several writing and instruction settings, the fixed termination rule matches or exceeds the full method" (W2); "Does the learned predictor retain substantial benefits over the termination-only rule on ... a more recent, stronger open model?" (Q1)
The termination rule is not a baseline that our method competes with. It is our method. Our contribution is a construction that exposes length information before generation while preserving the output distribution exactly, for any choice of candidate distribution (Theorem 4.1). The termination rule is the simplest such choice: no predictor and no training.
Seen this way, the result the reviewer points to is a positive one. On writing and instruction settings, our construction separates response lengths (for example by 1.6–2.0× for Mistral-7B) with the output distribution exactly preserved and without training anything. We should have presented it as a result and not as an ablation, and the revision will do so.
The learned predictor is for the settings where this simplest version is not enough. That is the case for reasoning models, including the larger and more recent ones the reviewer asks about (GSM8K, α = 0.5, 480 held-out prompts × 8 samples; ratio of long-regime to short-regime mean length):
Here the predictor supplies all of the separation, up to 70B. So the two versions cover different settings: a deployment uses the free one where it works and the predictor where it is needed.
On the predictor's "additional complexity and computational cost": it is a 5M-parameter head on the frozen model, trained on a CPU in minutes. At inference it adds a fixed amount of work per decoding step, which we quantify in point 8 of our reply to Reviewer khLQ.
2. "without quantifying how much the regime reduces prediction error or uncertainty beyond knowing the prompt, or demonstrating corresponding scheduling, latency, or resource benefits" (W2)
We have now measured both (new).
Uncertainty. Consider an oracle prompt-level predictor, which knows the true mean response length of every prompt. No predictor that sees only the prompt can do better. On GSM8K it is still off by hundreds of tokens, because the variation is within the prompt. Knowing our regime reduces that error:
(Errors are root-mean-square, in tokens.) In variance terms, the regime explains 24–34% of the length variance that remains once the prompt is known. This comes from a single bit that is available before the first token is generated.
Scheduling. We simulated two standard uses on our measured response lengths, again starting from the oracle prompt predictor. Under shortest-job-first scheduling, adding the regime cuts mean queueing delay by about 10% on these three workloads. In static batches of 16 sorted by predicted length, it raises the share of decode steps that do useful work from about 36% to 41–43%, which is 13–18% more throughput from the same hardware. On Qwen3-8B, where the regimes do not separate, both gains are zero, which shows they come from the length signal.