acceptodds
Under review as a conference paper at ICLR 2027

RetryPrec: Precision as a Retry-Time Resource under an External Verifier

Abstract

A quantized language model is often served next to its BF16 original, and a task with a verifier lets a failed attempt be tried again. After a low-precision failure, should the next attempt stay in low precision or switch? The usual rule ranks the two copies by average accuracy and repeats the winner. However, a failure is a selection event: the problems it leaves open lean toward those on which the quantized copy is weak, and at our reference setting a BF16 attempt solves more problems per second from the second failure on. We show that exchanging one low-precision attempt for a high-precision one changes the solve rate by the mean gap times the fraction of problems still open, plus a covariance between staying unsolved and the precision gap. We propose RetryPrec, which runs one block of low-precision attempts and then one block of high-precision attempts, with the block lengths chosen on held-out problems by an unbiased estimate of the solve rate. One such schedule is optimal among the policies that react only to failures, and we derive how failures move the ratio of the two hazards. On twelve settings at matched wall clock RetryPrec reaches points, against for the fixed low-precision schedule and for the best of sixteen other strategies. A test registered before the experiments shows that mixing precisions adds little once each problem uses its better one, so most of the attainable gain lies in choosing a precision for each problem, which failures partly reveal. When the budget also charges the memory held by both checkpoints, the schedule stays in low precision.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
calibration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GURdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This MS questions whether, after a quantized language model fails a verifier-checked attempt, the next try should use the same low-precision copy or rather the BF16 original. Assuming independent retries and a perfect verifier, it demonstrates how failures leave behind problems on which the quantized copy tends to struggle, which would make BF16 increasingly worth considering even though the faster quantized copy initially solves more problems per second. The authors thus propose the strategy of trying the quantized copy a few times before switching to BF16. Their method, namely RetryPrec, chooses how many attempts each copy gets by testing every allocation that fits the time budget on a separate set of calibration problems. Across several models and related tasks, it reports modest gains over repeatedly using the quantized copy and the strongest competing strategy. The authors also acknowledge that, once the memory needed to keep both copies loaded is counted as a cost, the method chooses low precision throughout and the benefit of switching eventually disappears.

Critical strengths

The manuscript points out a critical issue in how retries are handled when a quantized model runs next to its original BF16 version. When a retry fails it, then it typically leaves behind a set of problems, especially ones that the quantized model has trouble with. Because of this, just comparing the two models by accuracy can largely miss how much better BF16 really is. The paper backs up this idea with a clear formula that demonstrates the benefit of using BF16 depends on whether it does perform better on the problems the quantized model can’t properly solve. The set of measurements also reveal differences in how each model performs on problems/differences that average accuracy alone doesn’t show. I think the strongest part of the ÖS is how it bridges the math to what’s seen in practice. It gives a plausible reason, for using a simple retry plan and helps explain when switching to BF16 is worth the extra effort.

Critical weaknesses

The core comparison of the paper does not automatically demonstrate an clear advantage over a straightforward, more powerful baseline that is already provided in the appendix. To elaborate, while in Table 31, sampling the quantized version only at temperature 1.3 beats RetryPrec in all four task families in Qwen2.5-32B, with a smaller mean stopping time and a single loaded checkpoint, Table 1 uses temperature 1.0 for the temperature baseline. This is totally critical since the supposed advantage of precision switching may well be offset by the improved sampling. The speculative decoding comparison should also contain an explanatory note i.e., it gets the same number of attempts and costs as regular BF16 in the core comparison but cheaper attempts when combined with RetryPrec in Table 33.

In addition, the algorithmic contribution of the MS beyond prior work on allocation and cascading techniques is somewhat limited. In terms of allocation, OSCA (Zhang et al., 2025) already knows how to allocate a sampling budget over inference configurations based on training problems. Similarly, RetryPrec does this for two levels of precision, with the cheap-to-expensive schedule being part of the same policy class as the FrugalGPT (Chen et al., 2024b) baseline in the paper. However, the comparator for the OSCA framework is limited to the same precision level configurations only, rather than being able to make the same low-precision/BF16 choices that RetryPrec makes. The covariance and hazard analyses are also insightful, and yet the experiments do not seem to establish the existence of an algorithmic improvement over the prior work. The distinction here is important: While an insightful analysis can constitute a contribution, it doesn’t automatically imply an improvement over prior work.

Questions for authors
  1. Why does the main table compare against temperature only at T=1.0T = 1.0 when Table 31 shows the quantized model at T=1.3T = 1.3 T=1.3 outperforming RetryPrec across all four task families?
  2. Does RetryPrec still lead when the temperature is tuned for both copies? The response indeed would determine whether the method has persuasive practical implications.
  3. How does allowing a second switch change “the solve rate” in Table 32 when the chosen pair and its reported costs remain identical, and why is speculative decoding charged the full BF16 cost in Tables 1 and 21?
  4. Where do the evaluation and calibration problems come from?, and also how is the verifier defined for the multiple-choice and numeric-answer task families?
1 comment
Reviewer_GURdOpenReview ↗

Response to Author's Rebuttal

I’d like to thank the authors for the detailed rebuttal and the effort put into revising the MS. I went through the updated paper. The revised experiments and additional explanations have certainly improved the paper. But still, a few reservations remain in my opinion. Let me elucidate.

  1. The updated results in Table 36 address my earlier concern, to a large extent, about the under-tuned baselines. At (T=1.3), RetryPrec achieves 82.62%, compared with 80.54% for fixed low precision and 81.52% for the strongest alternative. Maybe not a major breakthrough, these improvements are supported by the reported confidence intervals. This is a useful clarification to my account, although the 1.10-point advantage over the strongest competitor remains relatively modest.

  2. I'm still a bit hesitant about the novelty side of things. I appreciate the new controlled comparison with OSCA in Table 41. The improvements of 1.31-1.79 points under limited calibration are definitely encouraging. However, with (k=64), the difference drops to just 0.08 points (79.46 versus 79.38), with the confidence interval including zero. To my account, this suggests that the main algorithmic benefit comes from a better estimator under limited calibration rather than a fundamentally different allocation strategy, which was the main novelty point. The covariance and hazard analyses are also interesting, but I am still not fully convinced that the algorithmic contribution is sufficiently distinct from existing approaches.

  3. The additional memory requirements remain as a concern. At the reference setting, keeping both checkpoints resident requires about 86.5 GB compared with 19.1 GB for low precision alone. Under the constraints of memory-time budget, RetryPrec would fall back to using low precision throughout. I recognize that the authors discuss alternative deployment arrangements, but these results still suggest that the practical benefit depends quite strongly on the serving setup. It is also worth noting that calibration costs are excluded from the main budget.

  4. The revised accounting for speculative decoding and additional switches is mostly satisfactory. The explanations of the dataset composition, difficulty tiers, and verification procedures are also much clearer now. One remaining inquiry is how well the reported gains would transfer to naturally distributed workloads, given the intentional balancing of problem difficulty and the assumptions of independent retries and the assumption of a perfect verifier.

Overall, I find the revised manuscript better codified, and some of my earlier experimental concerns have been adequately addressed. Nevertheless, my main reservation regarding the degree of algorithmic novelty and the practical significance of the improvements still remains. While my assessment of the technical presentation has improved, I do not think the revision changes the overall contribution sufficiently to warrant a change in my original recommendation.

Reviewer_ts2EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether a failed quantized model's attempt, which can be verified from the additional verifier, should be followed by same quantized model's attempt or a switch to the full-precision model. Rather than simply suggesting that a high-precision model should be used after a failure, the paper observes that the model that is efficient across all problems may differ from the one that is efficient on problems remaining unsolved after repeated failures. Therefore, using the joint law of the two per-problem success probablities, it analyzes how failures change the conditional success probabilities of the two models. Based on this analysis, which allocates retry attempts between the two precisions under a given budget. On twelve settings at matched wall clock RetryPrec achieves higher average score than the other serving baselines.

Critical strengths
  • The defined problem and analysis is interesting. I think it can deliver a good insight to anyone who want to provide high-capacity solution with a lower inference cost.
  • The evalation scope is wide and comprehensive. The paper compare its method across various model sizes and tasks.
Critical weaknesses
  • Reasoning models are highly depend on the sampling for each iterations. And this sampling introduces high randomnesses in terms of final pass or fail. These randomnesses can make the scheduling or analysis challenging. At the same time, the proposed method doesn't show the clear gap compared to the other methods. It would be beneficial to justify that the experimental design was adequately controlled for sampling randomness, and the results are reliable under this randomness issue.
  • Although the quantized model introduces lower cost per iteration, the quantized model can generate longer reasoning trajectory due to internal reasoning failure, therefore the total cost can be larger than the full-precision model inferences. Can the framework adequately handle such cases?
  • I think the decoding hyperparameter (e.g., higher temperature can increase randomness during decoding) can largely affect for the schedule or the end performance. It would be beneficial to discuss the impact of the decoding hyperparameters.
  • While considering the scenario that both baseline and quantized model are serving with verifier, we need to consider more deployable options. For example, how does the propsosed method compare with a naive option, generating multiple candidates in parallel then verifying the answers in terms of accuracy and latency?
Questions for authors

Please see Weaknesses.

Reviewer_unomOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes an adaptive low- to high-precision model-switch which offers competitive performances across various domains in wall-clock time compared to existing baselines.

Critical strengths
  • The proposed method is extremely simple: switch from low- to high-precision models when low-precision model faces the failures in one block of attempts.
    • This paper provides the detailed theoretical analysis on how the gains from switching are derived, helping understanding how switching works.
Critical weaknesses
  • Main concern. The overall writing of the paper includes numerous expressions or phrases that are written with no clear definition or chosen with ambiguous words, which make hard to follow what the authors try to deliver as the main contribution of the paper.
    • Ex) Survival analysis calls this a selection effect: in a population of mixed risks the event rate among the survivors falls over time, because the units with the highest rates leave first (Vaupel et al., 1979). -> what is event rate? what is unit?
    • Ex) At our reference setting, Qwen2.5-32B with GPTQ-W4 on code, the yield per second of one more quantized attempt -> what is yield here? what does it mean by “one more quantized attempt”?
    • Ex) Exchanging one low-precision attempt for a high-precision one changes this by the mean gap times the fraction still open, plus a covariance between staying unsolved and the precision gap -> what does “mean gap times the fraction still open” mean?
    • Ex) “The covariance admits two readings:”
  • I’m not convinced how impactful the method this paper delivers. While it outperforms the baselines in wall-clock time, it is only available when additional memory charges is necessary, where its contribution is highly defeated by other baseline. Additional core contribution, such as more adaptive switch, that significantly improves efficiency in both time and memory, should be included to be more impactful.
  • Calibration is not free. It should be considered as the overhead compute for the method to be fairly compared with baselines.
Questions for authors

Please see weaknesses.

1 comment
AuthorsOpenReview ↗

Your three concerns, and where the revised paper answers each

Thank you for your review. Your four examples showed us exactly where the opening was hard to read, and your request for a more adaptive switch led to RetryPrec-Adapt. As the discussion closes, we go through your three concerns and show where the revised paper answers each.

1. Writing, your main concern: the sentences you quoted are rewritten. The abstract and Section 1 now use plain machine-learning terms, marked in blue. The selection effect reads: "across test problems of varying difficulty, the conditional pass probability on remaining unsolved problems drops after each failed retry, because easier problems are solved and exit first". The yield reads: "the pass rate per second of one additional quantized retry, defined as its conditional pass probability on currently unsolved problems divided by its latency". The exchange now begins "replacing one quantized retry with a BF16 retry changes the overall solve rate by two terms" and states both terms in words, and "The covariance admits two readings" now reads "A positive second term can arise in two ways". Appendix A lists every symbol and term.

2. Impact under a memory charge, and the more adaptive switch you asked for.

  • At matched wall clock, RetryPrec beats fixed low precision in all 12 settings: by 2.83 points at the paper's temperature and by 2.08 at the tuned temperature 1.3, with 95% intervals above zero (Table 36).
  • When both copies must stay resident for the whole run, the rule returns pure low precision and loses nothing. The abstract states this boundary.
  • Once the copies need not share memory for the whole run, through staged swap-in or separate pools, the switch gains 0.44 to 0.54 points under the memory-time budget (Table 37).
  • RetryPrec-Adapt switches each problem on its own. It reads the entropy of the failed attempt, which decoding has already computed, and needs no extra calls, no router and no extra memory. It adds 0.52 points at wall clock, raises the share of the per-problem headroom it collects from 41% to 50%, and gains 0.61 to 0.81 points over single precision under the memory-time budget (Table 37).

3. Calibration is not free: it is now charged. Table 38 deducts the calibration time from RetryPrec's own budget, while every comparator keeps its full budget. Spread over 25,000 served problems, under 7 hours at one query per second, RetryPrec still leads every comparator without calibration on the 12-setting average; over 100,000 problems, the charged averages equal the uncharged ones. The calibration is small by design, 128 problems with 8 attempts each, because the score is unbiased for every block of up to 8 attempts.

4. What the evidence rests on. Every comparison uses 64 attempts per problem and configuration, with paired bootstrap intervals over problems. The gain over fixed low precision holds on Qwen2.5 at 7B, 32B and 72B, on Llama-3.1-8B and on R1-Distill-Qwen-32B, at matched temperatures and with 95% intervals above zero (Table 36). Your review credits "the detailed theoretical analysis on how the gains from switching are derived"; Proposition 4 adds that no policy reacting only to failures can beat the best one-switch schedule.

5. Your scores. None of the three concerns questions the correctness of a result: the first is about wording, the second about a deployment boundary that the paper now states, and the third about accounting, which Table 38 now does. Your review rated soundness 1, for concerns that "do not appear easy to address", and presentation 1. The ICLR guidelines ask reviewers to update the review with what the discussion and the revision brought.If one point still falls short, please tell us which; the discussion closes today, and we will answer at once.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.