- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This MS questions whether, after a quantized language model fails a verifier-checked attempt, the next try should use the same low-precision copy or rather the BF16 original. Assuming independent retries and a perfect verifier, it demonstrates how failures leave behind problems on which the quantized copy tends to struggle, which would make BF16 increasingly worth considering even though the faster quantized copy initially solves more problems per second. The authors thus propose the strategy of trying the quantized copy a few times before switching to BF16. Their method, namely RetryPrec, chooses how many attempts each copy gets by testing every allocation that fits the time budget on a separate set of calibration problems. Across several models and related tasks, it reports modest gains over repeatedly using the quantized copy and the strongest competing strategy. The authors also acknowledge that, once the memory needed to keep both copies loaded is counted as a cost, the method chooses low precision throughout and the benefit of switching eventually disappears.
Critical strengths
The manuscript points out a critical issue in how retries are handled when a quantized model runs next to its original BF16 version. When a retry fails it, then it typically leaves behind a set of problems, especially ones that the quantized model has trouble with. Because of this, just comparing the two models by accuracy can largely miss how much better BF16 really is. The paper backs up this idea with a clear formula that demonstrates the benefit of using BF16 depends on whether it does perform better on the problems the quantized model can’t properly solve. The set of measurements also reveal differences in how each model performs on problems/differences that average accuracy alone doesn’t show. I think the strongest part of the ÖS is how it bridges the math to what’s seen in practice. It gives a plausible reason, for using a simple retry plan and helps explain when switching to BF16 is worth the extra effort.
Critical weaknesses
The core comparison of the paper does not automatically demonstrate an clear advantage over a straightforward, more powerful baseline that is already provided in the appendix. To elaborate, while in Table 31, sampling the quantized version only at temperature 1.3 beats RetryPrec in all four task families in Qwen2.5-32B, with a smaller mean stopping time and a single loaded checkpoint, Table 1 uses temperature 1.0 for the temperature baseline. This is totally critical since the supposed advantage of precision switching may well be offset by the improved sampling. The speculative decoding comparison should also contain an explanatory note i.e., it gets the same number of attempts and costs as regular BF16 in the core comparison but cheaper attempts when combined with RetryPrec in Table 33.
In addition, the algorithmic contribution of the MS beyond prior work on allocation and cascading techniques is somewhat limited. In terms of allocation, OSCA (Zhang et al., 2025) already knows how to allocate a sampling budget over inference configurations based on training problems. Similarly, RetryPrec does this for two levels of precision, with the cheap-to-expensive schedule being part of the same policy class as the FrugalGPT (Chen et al., 2024b) baseline in the paper. However, the comparator for the OSCA framework is limited to the same precision level configurations only, rather than being able to make the same low-precision/BF16 choices that RetryPrec makes. The covariance and hazard analyses are also insightful, and yet the experiments do not seem to establish the existence of an algorithmic improvement over the prior work. The distinction here is important: While an insightful analysis can constitute a contribution, it doesn’t automatically imply an improvement over prior work.
Questions for authors
- Why does the main table compare against temperature only at when Table 31 shows the quantized model at T=1.3 outperforming RetryPrec across all four task families?
- Does RetryPrec still lead when the temperature is tuned for both copies? The response indeed would determine whether the method has persuasive practical implications.
- How does allowing a second switch change “the solve rate” in Table 32 when the chosen pair and its reported costs remain identical, and why is speculative decoding charged the full BF16 cost in Tables 1 and 21?
- Where do the evaluation and calibration problems come from?, and also how is the verifier defined for the multiple-choice and numeric-answer task families?
Response to Author's Rebuttal
I’d like to thank the authors for the detailed rebuttal and the effort put into revising the MS. I went through the updated paper. The revised experiments and additional explanations have certainly improved the paper. But still, a few reservations remain in my opinion. Let me elucidate.
The updated results in Table 36 address my earlier concern, to a large extent, about the under-tuned baselines. At (T=1.3), RetryPrec achieves 82.62%, compared with 80.54% for fixed low precision and 81.52% for the strongest alternative. Maybe not a major breakthrough, these improvements are supported by the reported confidence intervals. This is a useful clarification to my account, although the 1.10-point advantage over the strongest competitor remains relatively modest.
I'm still a bit hesitant about the novelty side of things. I appreciate the new controlled comparison with OSCA in Table 41. The improvements of 1.31-1.79 points under limited calibration are definitely encouraging. However, with (k=64), the difference drops to just 0.08 points (79.46 versus 79.38), with the confidence interval including zero. To my account, this suggests that the main algorithmic benefit comes from a better estimator under limited calibration rather than a fundamentally different allocation strategy, which was the main novelty point. The covariance and hazard analyses are also interesting, but I am still not fully convinced that the algorithmic contribution is sufficiently distinct from existing approaches.
The additional memory requirements remain as a concern. At the reference setting, keeping both checkpoints resident requires about 86.5 GB compared with 19.1 GB for low precision alone. Under the constraints of memory-time budget, RetryPrec would fall back to using low precision throughout. I recognize that the authors discuss alternative deployment arrangements, but these results still suggest that the practical benefit depends quite strongly on the serving setup. It is also worth noting that calibration costs are excluded from the main budget.
The revised accounting for speculative decoding and additional switches is mostly satisfactory. The explanations of the dataset composition, difficulty tiers, and verification procedures are also much clearer now. One remaining inquiry is how well the reported gains would transfer to naturally distributed workloads, given the intentional balancing of problem difficulty and the assumptions of independent retries and the assumption of a perfect verifier.
Overall, I find the revised manuscript better codified, and some of my earlier experimental concerns have been adequately addressed. Nevertheless, my main reservation regarding the degree of algorithmic novelty and the practical significance of the improvements still remains. While my assessment of the technical presentation has improved, I do not think the revision changes the overall contribution sufficiently to warrant a change in my original recommendation.