- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper asks whether decode budgets and prompt hardness labels measured on a BF16 model can be reused after quantization. It proposes an audit protocol that controls for truncation, sampling variability, and global shifts in accuracy. On Qwen3.5-4B with GPTQ-W4, reusing the BF16 budget incurs little average accuracy loss, whereas hardness labels change substantially. Hardness-based SFT shows no detectable advantage over matched random selection. The authors frame these findings as a case study rather than a general law.
Critical strengths
- The paper addresses a useful deployment question. The audit protocol, particularly its explicit treatment of confounding factors, is its strongest contribution.
- The experimental controls are thoughtful. Paired decoding, same-configuration repeats, size-matched comparisons, and forward simulation help distinguish quantization effects from sampling noise and changes in overall accuracy.
- The authors clearly report negative results and limitations, including earlier findings that did not replicate, grading corrections, and an inconclusive RTN analysis. These disclosures help readers assess the evidence.
Critical weaknesses
- The main findings rely on only one model and only one primary quantizer. RTN provides limited independent confirmation because most of its hard set contains truncated traces. The simulator is also miscalibrated on the untruncated subset. I believe that a second quantizer evaluated under cleaner conditions would strengthen the conclusions.
- I found the paper difficult to follow. Many paragraphs present several numerical comparisons alongside dense cross-references, and I often had to reread them to identify the main finding. Clearer plots, charts, tables and brief descriptions of the underlying trends would make the argument easier to understand.
- The budget-transfer result only covers a boundary optimum. BF16 selects the longest budget in all six seeds, so the study does not establish whether an interior optimum survives quantization. This limits the practical scope of the budget recommendation.
- The calibration setup raises concerns about how broadly the findings apply. In my experience, matching calibration sequences to the downstream length and format can affect quantization quality. Here, short multiple-choice calibration data differ considerably from the long reasoning tasks being studied. The observed degradation may therefore depend on this particular recipe. Any item overlap between calibration data and MMLU-Pro evaluation also needs clarification.
Questions for authors
- What aggregation rules, and decoding protocols were used for Table 13? Its GPQA values, including BF16 mid-budget accuracy of 76.26%, do not match Table 6. Please explain the relationship between the two tables.
- The BF16–W4 Jaccard overlap is 0.83 on the unrestricted OpenR1 pool but 0.414 on the teacher-correct subset. Does the higher full-pool overlap mainly reflect prompts that both models fail to solve? Could both values be reported together in the main text, with a discussion of their implications?
- Could the core audit be repeated using calibration sequences better matched to the downstream reasoning tasks in length and format? With other GPTQ settings held fixed, do the main conclusions remain unchanged? Please also clarify whether calibration and evaluation contain overlapping items.