acceptodds
Under review as a conference paper at ICLR 2027

Budgets Transfer, Hardness Labels Do Not: Auditing What Survives Quantization of Reasoning Models

Abstract

Reasoning models are usually quantized just before deployment, but two settings that guide their use are measured earlier, on the full-precision (BF16) model: the decode budget, which caps how many tokens the model may generate, and prompt hardness labels, which decide which prompts receive extra training. We ask whether these BF16 measurements can be reused after quantization. A direct comparison is misleading, because a longer budget can look better only because fewer answers are cut off, and because two runs of the same model already disagree on which prompts are hard. We therefore check truncation before comparing budgets, and compare every cross-precision change in the hard set with an independent rerun of the same model. On Qwen3.5-4B with 4-bit GPTQ, the two settings behave differently. The budget can be reused: serving the 4-bit model at the budget chosen on BF16 changes GPQA accuracy by −0.59 points on average over six seeds (95% interval [−1.94, +0.76]). Hardness labels cannot. The BF16 and 4-bit hard sets overlap by 0.414 (Jaccard), well below the 0.593 between two BF16 runs. The lower overall accuracy of the 4-bit model does not explain the gap: shifting and rescaling the BF16 scores to match it still predicts an overlap of 0.523. Quantization changes which prompts are hard, not only how many. A second quantizer, round-to-nearest, shows the same mismatch. The practical rule is to inherit the budget from BF16 but remeasure hardness on the deployed model, and to test any hardness-based training against a random control, since in our runs it gives no detectable gain.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Quantization, Test-Time Scaling, Metadata Transfer

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_b6dNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper asks whether decode budgets and prompt hardness labels measured on a BF16 model can be reused after quantization. It proposes an audit protocol that controls for truncation, sampling variability, and global shifts in accuracy. On Qwen3.5-4B with GPTQ-W4, reusing the BF16 budget incurs little average accuracy loss, whereas hardness labels change substantially. Hardness-based SFT shows no detectable advantage over matched random selection. The authors frame these findings as a case study rather than a general law.

Critical strengths
  1. The paper addresses a useful deployment question. The audit protocol, particularly its explicit treatment of confounding factors, is its strongest contribution.
  2. The experimental controls are thoughtful. Paired decoding, same-configuration repeats, size-matched comparisons, and forward simulation help distinguish quantization effects from sampling noise and changes in overall accuracy.
  3. The authors clearly report negative results and limitations, including earlier findings that did not replicate, grading corrections, and an inconclusive RTN analysis. These disclosures help readers assess the evidence.
Critical weaknesses
  1. The main findings rely on only one model and only one primary quantizer. RTN provides limited independent confirmation because most of its hard set contains truncated traces. The simulator is also miscalibrated on the untruncated subset. I believe that a second quantizer evaluated under cleaner conditions would strengthen the conclusions.
  2. I found the paper difficult to follow. Many paragraphs present several numerical comparisons alongside dense cross-references, and I often had to reread them to identify the main finding. Clearer plots, charts, tables and brief descriptions of the underlying trends would make the argument easier to understand.
  3. The budget-transfer result only covers a boundary optimum. BF16 selects the longest budget in all six seeds, so the study does not establish whether an interior optimum survives quantization. This limits the practical scope of the budget recommendation.
  4. The calibration setup raises concerns about how broadly the findings apply. In my experience, matching calibration sequences to the downstream length and format can affect quantization quality. Here, short multiple-choice calibration data differ considerably from the long reasoning tasks being studied. The observed degradation may therefore depend on this particular recipe. Any item overlap between calibration data and MMLU-Pro evaluation also needs clarification.
Questions for authors
  1. What aggregation rules, and decoding protocols were used for Table 13? Its GPQA values, including BF16 mid-budget accuracy of 76.26%, do not match Table 6. Please explain the relationship between the two tables.
  2. The BF16–W4 Jaccard overlap is 0.83 on the unrestricted OpenR1 pool but 0.414 on the teacher-correct subset. Does the higher full-pool overlap mainly reflect prompts that both models fail to solve? Could both values be reported together in the main text, with a discussion of their implications?
  3. Could the core audit be repeated using calibration sequences better matched to the downstream reasoning tasks in length and format? With other GPTQ settings held fixed, do the main conclusions remain unchanged? Please also clarify whether calibration and evaluation contain overlapping items.
Reviewer_ZBP6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the transferability of two settings from a full-precision model to its quantized version: the inference token cap and the prompt hardness labels used to select training data. By comparing generation outcomes across budgets for the BF16 and W4 models, they conclude that the selected budget largely transfers, evidenced by the small accuracy loss when W4 uses the budget selected on BF16. They also observe significantly lower hard-set overlap between BF16 and W4 than between BF16 and an independent BF16 repeat. To test whether this difference can be explained by a global correction, they conduct simulations using shift-and-scale transformations on the logit scale. They find that the tested transformations do not fully explain the hard-set disagreement, suggesting that quantization affects individual prompts differently rather than merely reducing overall accuracy.

Critical strengths
  1. The paper studies a practical transfer question. It investigates whether token budgets and hardness labels measured on BF16 can be reused after quantization, potentially avoiding repeated measurement costs.
  2. The hard-set comparison includes useful statistical controls. Independent same-precision repeats provide a reference for sampling variability, and the reported confidence intervals show that cross-precision disagreement exceeds that reference at the main threshold. The untruncated analysis also helps distinguish precision-related movement from budget-related truncation.
  3. The global-correction analysis goes beyond observing label disagreement. The logit-shift and shift-and-scale simulations test whether an overall performance change can explain the disagreement. Synthetic controls strengthen this analysis, although its fitting procedure needs more detail.
  4. The training experiment includes matched random controls. Matching subset size and repetition shows that the apparent advantage over full-pool training does not reliably come from selecting hard prompts. Reporting this negative result is useful.
Critical weaknesses
  1. The budget-transfer conclusion is supported by a limited budget grid. BF16 consistently selects 32K, the longest available budget. We do not know whether this is an optimal budget or simply the best among the tested choices. Longer budgets, such as 48K or 64K, could affect BF16 and W4 differently. The low inheritance cost therefore supports transfer within the tested grid, but provides limited evidence that optimal budgets transfer. The <5% truncation gate does not resolve this limitation. Truncated responses already count as wrong, and allowing more responses to finish is itself better serving. Why is an extra gate needed, especially when it does not show that the selected budget is optimal?
  2. The global-correction analysis is reasonable, but its conclusion is stronger than the evaluation establishes. Section 5.4 shows that the fitted shift-and-scale model does not reproduce the observed BF16–W4 overlap, but the fitting objective and procedure are insufficiently specified. It is therefore difficult to distinguish limitations of the assumed transformation from inadequate fitting. Moreover, they do not directly evaluate agreement between mapped BF16 labels and actual W4 labels against a W4-repeat reference. That comparison would more directly test whether corrected BF16 labels are usable. The current results support failure of the tested model, rather than ruling out global correction generally.
  3. The evidence has limited coverage across models and quantizers. The complete audit centers on Qwen3.5-4B with GPTQ-W4. Other models mainly provide budget checks, while RTN’s substantial truncation prevents a clean replication of the prompt-specific interpretation. Transfer behavior could depend on model size, family, and quantization method. The authors acknowledge this limitation, but it still restricts the scope of their practical recommendations.
  4. The presentation makes the methodology difficult to assess. Sections 3.2 and 5 introduce substantial notation without sufficiently explaining the workflow and estimation procedure. Distillation enters abruptly, the precision used for subsequent training is clarified mainly in the appendix, and the global-correction analysis is split across Sections 4 and 5. These issues obscure how the reported results support the claims.
  5. As a minor concern, the transferability analysis is limited to token budgets and hardness labels. I would like to see more settings studied, such as sampling temperature or top-p, to provide a richer picture of what transfers after quantization and whether any consistent patterns emerge.
Questions for authors
  1. Have you tested longer budgets, such as 48K or 64K, to determine whether BF16 and W4 still prefer the same cap beyond the current grid?
  2. What objective and procedure were used to fit the global transformations, and how did you verify that the fitting was adequate?
  3. How well do mapped BF16 hardness labels agree with actual W4 labels, compared with the agreement between two W4 runs, ideally on held-out prompts?
  4. Does the hardness-transfer result hold for other model sizes or families under conditions where truncation is controlled?
Reviewer_pKGwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether decoding budgets and prompt hardness labels determined on a full-precision reasoning model can be reused after quantization. The authors propose an audit protocol that controls for generation truncation, sampling variability, and overall accuracy degradation. They find that decoding budgets largely transfer in the configurations studied, whereas hardness labels change beyond the variability observed in repeated runs, and these changes cannot be adequately explained by a global shift and rescaling. Hard-prompt oversampling also shows no reliable advantage over matched random sampling. The paper therefore recommends remeasuring prompt hardness on the quantized model and validating hardness-based data allocation against matched random controls.

Critical strengths
  • The paper addresses a practically relevant question: whether budget choices and hardness labels can be reused after quantization.
  • The analysis goes beyond aggregate accuracy by combining hard-set overlap with forward simulation.
  • The paper is well organized and easy to follow.
Critical weaknesses
  • The main experiments are concentrated on a single model and quantization configuration. The additional models mainly provide budget comparisons, while substantial truncation prevents the second quantizer from independently validating the full set of conclusions. This limits the gegeneralizability of the findings.
  • The analysis shows that the tested location–scale corrections cannot fully explain the observed hard-set changes. However, more flexible monotonic recalibration methods are not evaluated, leaving open whether such methods could explain these changes without requiring full remeasurement.
  • The downstream benefit of remeasuring hardness remains unestablished. For instance, quantized-model hardness selection shows no reliable advantage over matched random controls.
Questions for authors
  • Can the main findings be replicated across additional model families and quantization methods?
  • Would retraining on the corrected hard sets demonstrate a reliable advantage over BF16-based selection or matched random controls?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.