acceptodds
Under review as a conference paper at ICLR 2027

Distribution-Preserving Response Length Separation for Language Models

Abstract

Response length is a major source of uncertainty in serving large language models. Under stochastic decoding, the same request can produce responses that differ several-fold in length: for prompts from a single dataset, within-prompt variance in response length can exceed between-prompt variance. This uncertainty undermines efficient resource management. Predicting length from the prompt cannot address within-prompt variation, while steering generation towards a target length changes the model's output distribution. We introduce **Length-Separated Sampling**, which divides generation into short and long regimes chosen at random before generation. We construct both regimes from the frozen target model so that their mixture is exactly the original distribution. The regime is therefore known before generation and informative about response length, while the responses users receive remain distributed exactly as under ordinary sampling. The long regime reliably generates longer responses than the short regime, with mean response length up to 2.8 times that of the short regime.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, inference-time sampling, speculative decoding, response length, length prediction, LLM serving, distribution-preserving sampling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_Pb2JOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper studies how to expose information about an upcoming response’s length before generation while preserving the target language model’s original output distribution. The authors propose Length-Separated Sampling, which combines an advance random choice of short or long regime, length-predictor guidance, per-token probability constraints, and regime-posterior updates to construct a marginal-preserving conditional sampling process. Experiments report length separation across 11 models and three datasets. The research question and proposed approach have merit, but the evaluation primarily relies on older datasets and small to medium-sized models, and the writing has clear shortcomings. Both the empirical evaluation and the presentation require further refinement.

Critical strengths

The paper poses an interesting and practically relevant research question: can information about response length be exposed before generation while preserving the language model’s original output distribution? Stochastic responses to the same prompt can differ substantially in length, leaving uncertainty that prompt-level length prediction alone cannot resolve. This question is worth investigating.

Critical weaknesses
  1. The experimental coverage and representativeness of the length distributions are insufficient to establish broad value for current reasoning and serving scenarios. Effectiveness experiments primarily rely on older, small to medium-sized models, and tasks are limited. The most convincing gains attributable to the learned predictor are concentrated on relatively simple GSM8K problems, without validation on complex, extended reasoning.

  2. In several writing and instruction settings, the fixed termination rule matches or exceeds the full method. The paper primarily reports ratios of regime mean lengths without quantifying how much the regime reduces prediction error or uncertainty beyond knowing the prompt, or demonstrating corresponding scheduling, latency, or resource benefits. It therefore remains unclear how much practical value the learned predictor provides relative to its additional complexity and computational cost.

  3. The writing and technical exposition need substantial refinement. For example, several citations use narrative formatting without appropriate parentheses when serving as sentence-final references.

Questions for authors
  1. Does the learned predictor retain substantial benefits over the termination-only rule on a more challenging reasoning task and a more recent, stronger open model? What primarily limits performance in settings such as Open-RS2, where the oracle indicates considerable available separation but the method achieves little?

  2. What is the maximum generation length, and what fraction of responses reaches this cap, particularly for Qwen3-8B? If a substantial fraction is truncated, the cap could suppress within-prompt length variation and partly explain the reported Qwen3-8B exception. Do the low variance and weak separation persist under a larger generation budget?

7 comments
AuthorsOpenReview ↗

Response to Reviewer Pb2J (1/2): termination rule, larger models, practical benefit

We thank the reviewer for the review. We answer each point in turn. Experiments marked (new) were run for this response; we are adding them to the paper and will upload the revised PDF during the discussion period.

1. "In several writing and instruction settings, the fixed termination rule matches or exceeds the full method" (W2); "Does the learned predictor retain substantial benefits over the termination-only rule on ... a more recent, stronger open model?" (Q1)

The termination rule is not a baseline that our method competes with. It is our method. Our contribution is a construction that exposes length information before generation while preserving the output distribution exactly, for any choice of candidate distribution (Theorem 4.1). The termination rule is the simplest such choice: no predictor and no training.

Seen this way, the result the reviewer points to is a positive one. On writing and instruction settings, our construction separates response lengths (for example by 1.6–2.0× for Mistral-7B) with the output distribution exactly preserved and without training anything. We should have presented it as a result and not as an ablation, and the revision will do so.

The learned predictor is for the settings where this simplest version is not enough. That is the case for reasoning models, including the larger and more recent ones the reviewer asks about (GSM8K, α = 0.5, 480 held-out prompts × 8 samples; ratio of long-regime to short-regime mean length):

modelour method, no predictorour method, with predictor
AceReason-7B1.012.11
R1-Llama-8B1.001.86
R1-Qwen-14B1.001.58
AceReason-Nemotron-14B (new)0.991.71
R1-Distill-Qwen-32B (new)0.981.37
R1-Distill-Llama-70B (new)1.011.52

Here the predictor supplies all of the separation, up to 70B. So the two versions cover different settings: a deployment uses the free one where it works and the predictor where it is needed.

On the predictor's "additional complexity and computational cost": it is a 5M-parameter head on the frozen model, trained on a CPU in minutes. At inference it adds a fixed amount of work per decoding step, which we quantify in point 8 of our reply to Reviewer khLQ.

2. "without quantifying how much the regime reduces prediction error or uncertainty beyond knowing the prompt, or demonstrating corresponding scheduling, latency, or resource benefits" (W2)

We have now measured both (new).

Uncertainty. Consider an oracle prompt-level predictor, which knows the true mean response length of every prompt. No predictor that sees only the prompt can do better. On GSM8K it is still off by hundreds of tokens, because the variation is within the prompt. Knowing our regime reduces that error:

GSM8K, α = 0.5mean response lengtherror of the oracle prompt predictorerror when the regime is also knownreduction
AceReason-7B1,2841,01682419%
R1-Llama-8B75564855215%
R1-Qwen-14B59244138513%

(Errors are root-mean-square, in tokens.) In variance terms, the regime explains 24–34% of the length variance that remains once the prompt is known. This comes from a single bit that is available before the first token is generated.

Scheduling. We simulated two standard uses on our measured response lengths, again starting from the oracle prompt predictor. Under shortest-job-first scheduling, adding the regime cuts mean queueing delay by about 10% on these three workloads. In static batches of 16 sorted by predicted length, it raises the share of decode steps that do useful work from about 36% to 41–43%, which is 13–18% more throughput from the same hardware. On Qwen3-8B, where the regimes do not separate, both gains are zero, which shows they come from the length signal.

AuthorsOpenReview ↗

Response to Reviewer Pb2J (2/2): generation caps, harder tasks, Open-RS2, writing

3. "If a substantial fraction is truncated, the cap could suppress within-prompt length variation and partly explain the reported Qwen3-8B exception" (Q2)

We tested this and it is not the explanation. Generation caps are 2,048–6,144 tokens depending on the model (the revision will state them, with truncation rates). Only 1.3–1.4% of Qwen3-8B's GSM8K responses reach the cap. We then raised its cap to 16,384, so that no response is truncated (new). Within-prompt variation is 17% of its total length variance, unchanged from 18% at the lower cap, against 47–48% for AceReason-7B and Open-RS3 at the same cap. The best achievable split also stays low (1.45, against 3.1–3.4 for those two models). Qwen3-8B simply varies little from sample to sample, so there is little for any method to separate.

4. "older, small to medium-sized models ... without validation on complex, extended reasoning" (W1); "a more challenging reasoning task" (Q1)

Models. The method needs a model's logits and hidden states, so it can only be evaluated on open-weight models. Within that constraint the evaluation is broad: eleven models in the paper, ten of them released in 2025, covering RL-trained reasoning models, distilled reasoning models and non-thinking models. With the three models added here it now spans 1.5B to 70B parameters.

Tasks. GSM8K problems are simple, but the responses are not short: these models produce 600–1,300 tokens of reasoning per problem, and it is the length of this reasoning that the method separates. To test a harder task we ran AceReason-7B on MATH Levels 1–2 (new), where its responses average 3,000 tokens. The method separates them by 1.14× [95% CI 1.12, 1.16], about 400 tokens per response. On MATH Level 5 our current predictor does not separate the regimes (1.02 and 1.05 on two models). The revision will state this as a limitation.

5. "What primarily limits performance in settings such as Open-RS2, where the oracle indicates considerable available separation but the method achieves little?" (Q1)

The limit is how our standard predictor was calibrated for this one model, not a lack of length signal. For Open-RS2 on GSM8K, the tilt built from the standard predictor pushed the regimes in the wrong direction on validation data. Our procedure detected this and fell back to using no predictor, so the Table 1 entry (1.09) is the no-predictor version.

The signal is there. We trained a variant of the predictor that outputs the probability of finishing early directly, so there is no tilt strength to calibrate. On the same model and dataset it reaches 1.96 on the Table 1 statistic, against an oracle of 2.49. Open-RS2 is therefore in line with the other reasoning models once the calibration problem is removed. The revision will report this variant.

6. "The writing and technical exposition need substantial refinement" (W3)

The citation formatting is fixed throughout. Section 4 will present a single construction, with the accept/reject alternative moved to the appendix (also requested by Reviewer 92Cw). We would welcome pointers to any other passages.

We hope these answers and the new experiments address the reviewer's concerns. If any remain, we would be grateful if the reviewer could say which, so that we can address them during the discussion period.

Thank you for the detailed response and the additional experiments on larger models. I also appreciate the improvements to the paper’s clarity and presentation. These results help address my concern about model coverage. However, I still see a substantial gap between the current evaluation and practical deployment in challenging reasoning settings. The positive results on MATH are limited to Levels 1–2, while the reported lack of separation on Level 5 raises concerns about effectiveness on more demanding problems, such as AIME-level questions.

I have increased my score to reflect the additional evidence and improved presentation. However, while I appreciate the acknowledgment of this limitation, I would need stronger validation on challenging reasoning workloads representative of practical use to support a higher score.

Reviewer_92CwOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents an approach for separating groups of responses to a prompt into groups of different lengths, while maintaining the exact distribution of responses, with the goal of improving serving efficiency. The approach is to first sample a Bernoulli for whether the response is in the short or long group, then to sample tokens that are likelier to result in the final completion being respectively short or long (via a conditional length prediction model). The paper evaluates this approach across a range of models and datasets, finding that indeed generations from the short group are generally shorter than generations from the long group.

Critical strengths
  • The paper highlights an interesting challenge, predicting and controlling the length of different completions to a given prompt.
  • To the best of my knowledge, the proposed method is novel
  • The method and the evaluation seem technically correct, and the evaluation convincingly demonstrates that length-separated sampling results in stratified lengths among result groups.
Critical weaknesses
  • The paper lacks some motivation for the precise scenarios in which this flavor of length-separated sampling leads to inference efficiency gains. In what types of workloads and deployments would this lead to actual efficiency gains?
  • The methodology section (4) is a bit hard to parse, since it describes multiple different potential approaches to marginal-matching sampling.
  • The paper doesn’t talk much about sensitivity of the results (e.g., mean separation ratio) to the quality of the remaining length predictor.
Questions for authors
  • In what types of workloads and deployments would this lead to actual efficiency gains?
  • How sensitive are the results to the quality of the remaining length predictor?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 92Cw: example workloads, new scheduling result, predictor sensitivity

We thank the reviewer for the positive and careful assessment. Both questions pointed to real gaps in the paper, and we believe it is stronger for the additions they prompted. We answer each point in turn below. Experiments marked (new) were run for this response; we are adding them to the paper and will upload the revised PDF during the discussion period.

1. "In what types of workloads and deployments would this lead to actual efficiency gains?" (W1, Q1)

The regime is useful to any system that must make a decision about a request before generating it, when the same prompt can produce short or long responses. Three examples:

  • Scheduling. A server that runs short jobs first reduces average waiting time. The regime tells it which requests are likely to be short.
  • Batching. A batch runs until its longest response finishes, so grouping requests of similar length wastes fewer decoding steps.
  • Sampling one prompt many times. Self-consistency, best-of-n and RL training methods such as GRPO draw many responses to the same prompt, and the group is only complete when the longest one finishes. A prompt-level length predictor gives every sample the same estimate. The regime distinguishes them, so the long ones can be grouped together or given more resources.

The paper did not demonstrate a concrete gain, and we have now added one (new). We simulated shortest-job-first scheduling on our measured response lengths. The baseline scheduler already knows each prompt's true mean response length, which is the most a prompt-level predictor could tell it. We then let it also see the regime:

workloadreduction in mean queueing delay from knowing the regime
R1-Qwen-14B × GSM8K10%
AceReason-7B × GSM8K10%
R1-Llama-8B × GSM8K9%
Qwen3-8B × GSM8K (regimes do not separate)0%

Knowing the regime cuts waiting time by about 10%, and by nothing on the control, which shows that the gain comes from the length signal. For batching, the same simulation gives 13–18% more throughput on these workloads. The gain follows the separation, so it is smaller on open-ended tasks, where our separation is smaller. The revision will add these examples to the introduction and the simulation as a new appendix.

2. "How sensitive are the results to the quality of the remaining length predictor?" (W3, Q2)

The guarantee does not depend on the predictor at all: Theorem 4.1 holds for any predictor, so even a bad one cannot change the output distribution. Only the amount of separation depends on it.

To show how much, we compared predictors of three different qualities on GSM8K (new). The weakest is no predictor. The middle one was trained only on story-writing and instruction data and never saw GSM8K. The best was trained on GSM8K.

predictorR1-Llama-8BAceReason-7B
none1.081.10
never saw GSM8K1.862.58
trained on GSM8K2.002.69

(Table 1 statistic; "none" is at the level of no separation.) Separation falls gradually as the predictor gets worse. A predictor applied to a domain it was never trained on still keeps most of it. The revision will add this as a new appendix section.

3. "The methodology section (4) is a bit hard to parse, since it describes multiple different potential approaches" (W2)

We agree. We included the accept/reject construction to show that the method is more general, but it is not used in any experiment and it distracts from the main line. Section 4 will present only the construction we use, and the more general accept/reject version will move to the appendix.

We thank the reviewer again. We would be glad to hear whether these additions address the concerns raised, and to answer any further questions.

Reviewer_khLQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper addresses within-prompt uncertainty in LLM response length by dividing generation into short and long regimes chosen at random prior to decoding. The method does not require pre-trained model fine-tuning and preserves the marginal distribution of the model's outputs. Evaluation is conducted on 11 models and 3 datasets.

Critical strengths
  • The paper touches on an important aspect of answer length prediction that could e.g. enable efficient routing. The method is mathematically sound and, as far as I was able to verify, correct.
  • The authors provide a broad (though limited to small models) evaluation of the method on 11 models and 3 datasets.
Critical weaknesses
  • The main motivation presented in the paper is to enable efficient routing, but this was not tested by the authors. It is unclear whether the practical benefits outweigh real-world application issues. Appendix C shows length predictor transfer is working quite well, but only on 2 out of 5 tested models and limited to the 3 evaluated datasets. It is not clear how to train a general-purpose length predictor that can be used for any downstream task. Moreover, it was not discussed what the within-prompt and between-prompt overlap is between short and long regimes, nor how misassigning responses to short/long batches affects performance.
  • The work lacks proper validation (comparison with different methods; measuring accuracy, perplexity, repetition metrics; real-world latency measurements). Furthermore, multiple claims in the paper are overstated (the most important are discussed below).

Validation and overstated claims:

  • The claim about metric guarantees is asserted theoretically but never empirically tested on any downstream metric. It should be verified whether downstream accuracy as well as text quality (too-early EOS, repetition, unnatural words affecting grammar or coherence) are preserved, given that the method modifies the sampling distribution at every decoding step.
  • The reported 1% increase in latency from the main paper is rather a best-case scenario than real latency, according to Appendix B. The 0.5 -> 0.25 ms per step drop is surprising and not discussed. The headline 1% number comes from a 32B model, which was not even evaluated with the proposed method.
  • Separation between long and short regimes is generally overstated. Reported separation is visible almost exclusively for the GSM8K dataset and not even for all models. For most models, the separation is below 2 for this dataset and generally slightly above 1 for others. Separation is generally the lowest for the largest 14B model, which raises the question of how it would behave for models with 70B parameters or more. Moreover, metrics close to 1 might be an effect of general variance, not the method itself. It would be good to measure the variance of the vanilla model in the reported settings as well as compare it to other methods.
Questions for authors
  • What are the real-world benefits (measured with numbers or mathematical derivations) of the method over standard routing? What is the within-prompt and between-prompt overlap of short and long regimes? How can the method be used in practice when the prompt domain is not known in advance?
  • Can the authors provide evaluation metrics for the tested models and datasets that would empirically confirm theoretical guarantees?
  • What is the latency evaluated over a (continuous) batching scenario, measured per prompt (instead of per batch) and then averaged, to provide a more realistic approximation?
  • Can the authors provide a baseline measuring the vanilla model's variance in the same settings as other models in Table 1, so that ratios near 1 can be distinguished from noise?
7 comments
AuthorsOpenReview ↗

Response to Reviewer khLQ (1/2): metric preservation, noise, larger models, practical benefit

We thank the reviewer for a careful and constructive review, and for taking the time to verify that the method is mathematically correct. The review's questions led us to run several new experiments, and we believe the paper is considerably stronger for them. We answer each point in turn below, and hope the reviewer will find that their concerns have been addressed. Experiments marked (new) were run for this response. We are adding all of them to the paper and will upload the revised PDF during the discussion period, with changes highlighted.

1. "The claim about metric guarantees is asserted theoretically but never empirically tested" (W2, Q2)

The guarantee is Theorem 4.1. Each regime does modify the sampling distribution, but a user receives the short regime with probability α and the long regime otherwise, and that mixture is exactly the original model's distribution. Accuracy, perplexity and repetition are functions of the output, so their distributions cannot change. We agree an empirical check is still worth having (new; GSM8K, 480 prompts × 8 samples):

modelaccuracy, ordinary samplingaccuracy, our methoddifference in points [95% CI]
R1-Llama-8B85.2%84.9%−0.3 [−1.4, +0.8]
DeepMath-1.5B88.1%87.4%−0.7 [−1.8, +0.3]
R1-Qwen-14B95.9%95.5%−0.4 [−0.9, +0.2]
AceReason-7B96.1%96.2%+0.1 [−0.5, +0.7]

No difference is distinguishable from zero (intervals are bootstrapped over prompts). The same holds for text quality on R1-Llama-8B: per-token negative log-likelihood (0.323 vs 0.327), repeated 4-grams (5.8% vs 5.9%), and the share of responses that reach a final answer (98.0% vs 97.9%). All three 95% intervals on the difference include zero, so there is no sign of early stopping.

2. "metrics close to 1 might be an effect of general variance" (W2, Q4)

Table 1 reports a 95% confidence interval for every entry, which measures the sampling variability of each ratio. The reviewer's baseline answers a further question, namely what value the statistic takes when there is no separation at all, and we have computed it (new): we shuffle the short/long labels within each prompt, which gives the value of our statistic when both sets of responses come from the same distribution. 88 of the 99 entries lie above the 97.5th percentile of this baseline. All 11 exceptions occur for Qwen3-8B, Open-RS2 on GSM8K, or two models on WritingPrompts; the revised table will mark them as not separating. The baseline also shows that our statistic sits slightly above 1.0 under no separation (median 1.04), and the revised caption will say so.

3. "Separation is generally the lowest for the largest 14B model, which raises the question of how it would behave for models with 70B parameters or more" (W2)

This is a fair question, and we have now run the full method on three additional models, including one at 32B and one at 70B (new; GSM8K, 480 held-out prompts × 8 samples, ratio of long-regime to short-regime mean length):

modelseparation
R1-Distill-Llama-70B (new)1.52
R1-Distill-Qwen-32B (new)1.37
R1-Qwen-14B (in the paper)1.58
AceReason-Nemotron-14B (new)1.71

The 70B model separates as well as the 14B models. The mixture matches ordinary sampling in every run.

4. "What are the real-world benefits ... over standard routing?" and "What is the ... overlap of short and long regimes?" (W1, Q1)

The regime gives a scheduler information about the length of this particular sample, which no prompt-level predictor can have. To measure its value we simulated shortest-job-first scheduling on our measured response lengths (new). The baseline is a perfect prompt-level predictor, which knows each prompt's true mean length. The scheduler sees only that mean and the regime, never the response's actual length. Mean queueing delay at load 0.9, in units of mean service time:

workloadperfect prompt predictor+ regimedelay reduction
R1-Qwen-14B × GSM8K4.443.9910%
AceReason-7B × GSM8K4.494.0610%
R1-Llama-8B × GSM8K4.684.259%
Qwen3-8B × GSM8K (regimes do not separate)3.283.280%

The regime cuts delay by about 10% relative to an oracle prompt-level predictor that knows each prompt's true mean length, and by nothing on the control where the regimes do not separate.

On overlap: the regimes do overlap. On these workloads a long-regime response is longer than a short-regime response to the same prompt with probability 0.62–0.71 (0.60–0.69 when the two responses come from different prompts; 0.5 would mean no information). The regime shifts the length distribution; it does not sort responses into disjoint bins. Each simulated request runs for its actual length, so the cost of this overlap is already included in the 10%.

AuthorsOpenReview ↗

Response to Reviewer khLQ (2/2): unknown domains, baselines, abstract wording, latency

5. "How can the method be used in practice when the prompt domain is not known in advance?" (W1, Q1)

Train one predictor on mixed data. We tested this (new; separation on GSM8K):

modelpredictor trained on GSM8Kone predictor for all three datasetspredictor that never saw GSM8K
R1-Llama-8B2.002.021.86
AceReason-7B2.692.722.58

On GSM8K, a single predictor trained on all three datasets does as well as one trained on GSM8K alone, and a predictor that never saw GSM8K keeps most of the separation. The worst case for an unfamiliar prompt is less separation: the output distribution is exact for any predictor. The predictor is also cheap, a small head on the frozen model trained on a CPU in minutes, and each model needs its own.

6. "comparison with different methods" (W2)

The two existing families are not alternatives to ours. Prompt-level predictors are complementary: point 4 measures our gain on top of a perfect one. Length-control methods change the output distribution, which is the one thing our problem setting rules out. Within our own framework the paper compares against the variant with no predictor (Appendix C.6) and against the best achievable split (Appendix C.1).

7. "Separation between long and short regimes is generally overstated" (W2)

We agree the abstract should be more precise, and thank the reviewer for pressing on this. Separation is largest on mathematical reasoning and smaller (typically 1.1–1.3×) on open-ended tasks, although point 2 shows these smaller separations are real. The revised abstract will read: "On mathematical reasoning, long-regime responses are 1.4–2.1 times as long as short-regime responses for most reasoning models, from 1.5B to 70B parameters; on open-ended tasks the separation is smaller."

8. Latency (W2, Q3)

The overhead is a fixed cost per decoding step, so it is largest for the small models we measured and shrinks as models grow. For the very large models that are served at scale it becomes a small fraction of the step. The method adds a fixed amount of work to each decoding step, a top-64 selection and a 5M-parameter head, whatever the size of the model. That is under 0.5 GFLOPs per token, against about 28 for a 14B model's forward pass and about 140 for a 70B model's: under 2% and under 0.5% of the computation, and less again for models of several hundred billion parameters.

Wall-clock time shows the same trend. At a fixed batch of 128 the method adds about 0.5 ms per step, which is 3.7% at 14B and 2.1–2.6% at 32B (Appendix B.4). The time added does not change; the step it is added to gets longer.

As requested, we also measured per-request latency under continuous batching (new; vLLM, Poisson arrivals, one H100). We did this on R1-Qwen-14B, the less favourable of the two sizes we timed. Our implementation adds 1–2 ms per step there, more than at a fixed batch because the batch changes from step to step and our compiled GPU graph is reused less often. Mean time per output token, averaged over requests, rises by 10% at an arrival rate that keeps about 32 requests in flight. At a fixed arrival rate close to saturation (about 128 in flight) it rises by 27%, because near capacity any slowdown is amplified by queueing; compared at the same number of requests in flight the cost is still about 10%. These figures are for a small model on one GPU and shrink with model size for the reason above.

The revised paper will report all of these measurements in place of "1%", which was the fixed-batch figure at 32B and batch 256. The reviewer is right that the lower per-step cost at batch 256 is unexplained; we have not identified its cause, and the appendix will say so. The 32B model was used for timing only, which is valid because the cost does not depend on the predictor's weights; the full method has now been run at 32B (point 3).

We thank the reviewer again. We would be glad to hear whether these changes address the concerns raised, and to answer any further questions.

Thank you for the detailed response and new results. The authors' response answers most of my questions, so I am ready to update my score.

One follow-up question first:

  • Do the simulated service times in point 4 include the per-token overhead reported in point 8?

Apart from this, a few smaller concerns remain:

  • Point 3 results confirm my worries, as separation within the Qwen family drops as the model size grows (1.68 at 7B, (α\alpha=0.5, from the paper) -> 1.58 at 14B -> 1.37 at 32B). A similar effect is visible for AceReason (2.78 at 7B -> 1.71 at 14B).
  • A comparison with other methods, even if they modify the output distribution, would strengthen the paper. If such a method does not affect the model's performance and provides bigger serving gains, it might still be used in practice.
Reviewer_xqHxOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

na

Critical strengths

na

Critical weaknesses

na

Questions for authors

na

1 comment
AuthorsOpenReview ↗

Response to Reviewer xqHx: a plain summary of the method

We thank the reviewer for their time and for the positive rating. We thought it might be helpful to give a short, plain summary of the paper, since the idea is actually fairly simple.

The problem. Ask a language model the same question twice and the two answers can differ several-fold in length. For anyone serving a model this is awkward, because decisions such as scheduling and batching have to be made before the answer exists. Predicting length from the prompt does not help here: the prompt is the same both times.

The idea. Before generating, we flip a coin. On heads, we nudge the model's sampling towards tokens that tend to lead to shorter answers. On tails, we nudge it the opposite way. So the coin tells us in advance whether the answer is likely to be on the short side or the long side.

The catch, and how we avoid it. Nudging a model normally changes what it says. We construct the two nudges so that they cancel exactly: averaged over the coin flip, every possible answer has precisely the probability the original model gives it (Theorem 4.1). A user cannot tell that anything was done, and accuracy and every other property of the output are unchanged. The serving system, which saw the coin, knows something about the length before the first token is produced.

What we find. On mathematical reasoning, "tails" answers are 1.4–2.1 times as long as "heads" answers for most reasoning models, and we have now confirmed this on models up to 70B parameters. The model itself is never retrained; the nudge comes from a small add-on predictor, and in some settings no predictor is needed at all.

We would be very happy to answer any questions, and to hear any thoughts the reviewer has on the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Discussion: Claims, Agreed Points and Limitations

We thank the reviewers for a careful and constructive discussion, which has made the submission considerably stronger. Two reviewers, Pb2J and khLQ, raised their ratings. This comment summarises where the paper now stands.

What the paper claims. A construction: two sampling regimes, built from the frozen model, whose mixture is exactly the model's own distribution (Theorem 4.1). A variable known before generation then carries information about response length, while every metric of the output is unchanged. To our knowledge this problem has not been posed before, and this is the first method for it. The abstract claims the construction and the measured separation, and nothing more.

What is agreed.

  • The construction is novel (92Cw) and correct (92Cw, khLQ). The theorem holds for any predictor and any tilt, so it covers every part of our implementation.
  • Our experiments are consistent with the guarantee. Accuracy, likelihood, repetition and completion rate show no difference from ordinary sampling.
  • The regime separates real responses. On GSM8K, long-regime responses are 1.4–2.1 times as long as short-regime responses for most reasoning models, on fourteen models from 1.5B to 70B. The regime explains 24–34% of the length variance that remains once the prompt is known, which no predictor reading the prompt can do.

What is contested: practical value (khLQ, Pb2J).

  • What we have. In simulation on measured response lengths, knowing the regime raises the throughput of length-sorted static batches by 10–18% beyond an oracle prompt-level predictor, before the cost of the method.
  • What we do not have. A measured net gain in a deployed system. Our current implementation adds between a few percent and 10% to the time per token, and under continuous batching that outweighs the simulated gain. Separation is smaller on open-ended tasks, on harder problems and for larger models.

The final version will state these as limitations, will not claim a deployed serving gain, and will say "GSM8K" in place of "mathematical reasoning" in the abstract.

Why we believe it is a contribution as it stands. The paper poses a new problem, gives the first solution with an exact guarantee, measures how well it works on fourteen models, and reports its cost and limits. Because the guarantee holds for any predictor and any implementation, improvements to either carry over without changing anything else in the paper.

We are grateful to the reviewers for their engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.