acceptodds
Under review as a conference paper at ICLR 2027

Fewer Bits, More Tries: Recovering Capability Under Memory Constraints

Abstract

Large language models are generally more capable than smaller ones, but their memory requirements can make local deployment difficult. We ask whether part of this capability gap can instead be recovered by running a smaller or more compressed model multiple times and selecting among its outputs using public or model-generated tests. Single-user local inference can leave hardware underutilized, allowing parallel candidate generation to trade additional test-time computation for accuracy at modest latency cost. Across four code-generation benchmarks, verified retry recovers substantial accuracy lost to model size or compression. A 7B Qwen model with eight parallel attempts essentially matches a 32B model on MBPP+ and exceeds it on the other three benchmarks, while using about less model memory, running up to faster, and using less energy. We observe the same tradeoff across model sizes, compression levels, NVIDIA GPUs, and Apple Silicon, although its systems benefit depends on hardware and recovery depth. These results show that model capacity can be traded for verified test-time computation when correct solutions remain reachable within a modest retry budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test-time compute, verification, code generation, memory-efficient inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7EYMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper asks when a smaller/compressed code model can replace a single attempt from a larger model under a memory budget. The smaller/compressed model runs K times in parallel, and a verifier selects one output. The motivation is that batch-size-1 decoding is bandwidth-bound, so extra parallel candidates cost little until the batch reaches the bandwidth upper bound.

The experiments use two model sets:

Qwen2.5-Coder at 1.5B–32B, in bf16, 8-bit, and 4-bit.

A controlled bf16 / 4-bit / 1-bit triplet of Qwen3.6-27B.

Paper tests on HumanEval+, MBPP+, LiveCodeBench, and BigCodeBench.

The paper explains that compression lowers pass@1 much more than oracle@K. Under self-verification, the paper splits failures into three sources: candidate generation, test discrimination, and aggregation. Apple Silicon measurements show that the upper bound K depends on the device.

Critical strengths
  1. A practical, well-posed question. Framing the trade-off as K_rec ≤ K_sys is a useful lens for deployment.

  2. The reliability/reachability distinction is supported. From 3B to 32B, pass@1 rises 7.7–16.3 pp on three benchmarks, while oracle@50 varies by only 1.3–2.4 pp.

Critical weaknesses
  1. There appear to be internal inconsistencies.

Table 1 vs. Table 21. The two tables give different values for the same configurations: Coder-7B K=8 on MBPP+: 77.8 vs. 78.0. Coder-7B K=8 on LiveCodeBench: 27.5 vs. 27.7. Coder-32B K=1 on HumanEval+: 85.1 vs. 85.3.

CodeT exceeds the oracle. In Table 5 (LiveCodeBench), CodeT@8 is 37.2 / 37.1 / 38.1%, which is higher than oracle@50 (35.4 / 34.3 / 34.3%) for all three checkpoints.

  1. All evidence comes from code generation. The title and abstract present a general trade-off, but all four benchmarks are code generation.

  2. Positioning relative to prior test-time scaling work could be sharpened. Several of the paper’s central observations are consistent with, and in part anticipated by, earlier work[1][2][3][4].

[1] Large Language Monkeys: Scaling Inference Compute with Repeated Sampling.

[2] Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Parameters for Reasoning.

[3] Not All Bits Are Equal: Scale-Dependent Memory Optimization Strategies for Reasoning Models.

[4] The Limits of Inference Scaling Through Resampling.

Questions for authors
  1. How can CodeT@8 exceed oracle@50 on LiveCodeBench in Table 5?

  2. Do the reliability-vs-reachability pattern and the K_rec ≤ K_sys hold on a math benchmark or other benchmarks?

2 comments
AuthorsOpenReview ↗

Response, part 1/2

We thank the reviewer for the careful reading and constructive feedback. Both inconsistencies are resolved in the revision, and we have added a transfer study on mathematical reasoning and a sharper comparison with prior work. All changes are marked in red in the revised PDF.

Reviewer: Table 1 vs. Table 21 give different values for the same configurations (77.8 vs. 78.0, 27.5 vs. 27.7, 85.1 vs. 85.3).

Response: Thank you for catching this. Accuracies at intermediate KK are Monte Carlo estimates over random KK-subsets of each fixed candidate pool, and the two tables had been filled from different estimation runs. In the revision, Tables 1 and 4 use a single set of estimates, with Coder-7B at K=8 reaching 78.0% on MBPP+ and 27.7% on LiveCodeBench. Table 21 now reports accuracy only at K=1K=1 and K=50K=50, where it is computed exactly from the full pool and matches pass@1 and exec@50 in Table 9; Coder-32B on HumanEval+ is therefore 85.1% at K=1K=1. Intermediate budgets are shown in Table 1 and Figure 1. Section 4.1 now describes the procedure: K=1K=1 and K=50K=50 use the full pool, and intermediate KK averages 120–300 random subsets per task (standard error at most 0.15 points).

The changes are at most 0.3 points and leave every conclusion unchanged. Appendix F.1 adds paired task-bootstrap intervals for the main comparisons: at K=8K=8, Coder-7B is significantly more accurate than Coder-32B one-shot on HumanEval+ (+4.4 [+1.0, +8.2] points) and statistically indistinguishable from it on MBPP+ (+0.1 [−2.9, +3.1]) and LiveCodeBench.

Reviewer: CodeT@8 (37.2 / 37.1 / 38.1%) exceeds oracle@50 (35.4 / 34.3 / 34.3%) on LiveCodeBench in Table 5. How is this possible?

Response: The reviewer is right that this cannot happen on a common candidate pool and task set. The printed CodeT values were computed on the 82 LiveCodeBench tasks for which all three checkpoints produced usable generated-test suites, while oracle@50 was computed on all 175 tasks, so the two columns were not comparable. We have removed these CodeT cells from Table 5.

To provide the comparison on a common basis, we generated fresh candidates for all 175 tasks with each checkpoint and computed CodeT and the oracle on the same candidates (Appendix F.9). CodeT@8 is 16.0%, 18.2%, and 19.1% for the one-bit, AWQ4, and bf16 checkpoints, against oracle@8 of 26.3%, 30.4%, and 30.8%. These values are now reported in the Table 5 caption and Appendix B.1: on LiveCodeBench, model-generated tests recover much less than public-test selection.

The public-test results for the one-bit model in Table 2 are unaffected: they use the full 175-task pool, where exec@50 = 34.9% is below oracle@50 = 35.4%. We have also clarified in Section 5 that the signature-group statistic sig-oracle@KK is a diagnostic of how much the test outcomes can resolve, while candidate coverage (oracle@KK) remains the upper bound.

Reviewer: All evidence comes from code generation. Do reliability-vs-reachability and Krec≤KsysK_{rec} \leq K_{sys} hold on math?

Response: We added a transfer study on MATH-500 (Appendix E) with Qwen2.5-Instruct 3B, 7B, and 32B in bf16 and the AWQ4 builds of 7B and 32B. Math has no executable tests, so we compare confidence, majority vote, and the Qwen2.5-Math-PRM-7B reward model, with coverage as the upper bound.

Both findings carry over. Shrinking the model lowers reliability but not reachability: pass@1 rises from 65.0% (3B) to 82.6% (32B), while coverage at K=100K=100 is 96.0%, 96.6%, and 97.0%. Quantization behaves the same way: 32B AWQ4 recovers the one-shot accuracy of its bf16 parent within two majority-voted attempts.

The verifier sets the recovery depth. For 7B → 32B, KrecK_{rec} is 3 under the oracle, 8 with the reward model (bootstrap interval [4, 8]), and 16 with majority vote, while confidence alone never reaches the target. Measured end to end with both models resident and scoring included, eight 7B attempts take 0.66× the time of one 32B attempt for a 32-problem batch and 0.36× for a single request, using 39–64 GB of device memory against 77–82 GB for the 32B model.

The criterion Krec≤KsysK_{rec} \leq K_{sys} also identifies where retry does not pay: for 3B → 7B, majority vote needs 16 attempts against a measured systems budget of 4, and for 3B → 32B the reward model needs 32 attempts against a budget of 16. We have accordingly scoped the main deployment claim to tasks with informative and affordable verification (Sections 1 and 6).

AuthorsOpenReview ↗

Response, part 2/2

Reviewer: Positioning relative to prior test-time scaling work [1]–[4] could be sharpened.

Response: We revised Section 2 and the contributions. Our selector is standard best-of-N. The question we study is different: whether a smaller or compressed model that fits in memory can replace a larger one under simultaneous memory, accuracy, and wall-time constraints, including the verification that makes the extra samples usable.

[1] and [2] budget test-time compute in FLOPs or dollars. In local decoding the cost is dominated by reading the weights, which parallel attempts share, so eight 7B attempts on MBPP+ take only 1.62× the generation time of one at 60 concurrent requests (1.73× for a single request), not 8×. Section 3 explains this gap between FLOP cost and wall time, and our measurements quantify it. The success condition in [2] corresponds to what we measure as finite-budget candidate coverage.

[3] studies memory-constrained inference choices without executable verification. With public tests, we find that even a one-bit 27B model with 3.8 GB of weights recovers the one-shot accuracy of its 14× larger bf16 parent within two to four attempts (Table 2), because quantization lowers reliability while leaving coverage nearly intact.

[4] shows that imperfect verifiers limit resampling. We characterize when this happens: public-test selection lets Coder-7B reach or exceed the Coder-32B one-shot accuracy on all four code benchmarks, whereas generated tests give an incorrect candidate the same outcomes as the best correct one on 15–28% of tasks (Appendix F.4). Our diagnostic in Section 5 separates candidate-generation, test-discrimination, and aggregation failures.

The revised contributions (Section 1) are organized around (i) the different effects of shrinking and quantization on one-shot reliability versus coverage; (ii) a measured, verifier-inclusive break-even in place of a FLOP-only model; and (iii) the separation of generation, test-discrimination, and aggregation failures. Coder-7B at K=8 generates 1.1–4.0× faster than Coder-32B one-shot and uses 1.0–3.8× less GPU energy (Table 1), and remains 1.3–1.8× faster for a single request once the public tests are run (Appendix F.2).

Reviewer_KtXFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper explores how to compensate for the loss of capability caused by shrinking models or reducing accuracy due to memory limitations by increasing inference attempts. The method is to generate multiple code candidates in parallel, and then filter the answers using publicly available tests or model generated tests. Four code benchmarks show that the 7B model can basically catch up with or exceed the single performance of the 32B model after eight attempts, reduce about three quarters of the model memory on the B200, and achieve 1.3 to 5.9 times acceleration. After quantization to 1 bit, the same 27B model only needs 2-4 attempts to restore the single precision of the original model.

Critical strengths
  1. The motivation behind the paper is quite clear: it seeks to compensate for a lack of capability through repeated attempts.
  2. The authors conducted extensive experiments on multiple benchmarks.
Critical weaknesses
  1. The concept of using subsampling to improve accuracy is relatively straightforward; the primary contribution of this work lies in achieving a simultaneous speedup. However, the evaluation was conducted on B200 GPUs and Apple devices equipped with 64 GiB of memory. Whether the computational and caching overheads introduced by additional sampling might negate these gains on more common consumer-grade GPUs, typically featuring 8–16 GB of VRAM, remains to be verified through further experimentation.

  2. The experiments cover only single-turn code generation. It remains unclear whether smaller models can match the performance of larger models on tasks requiring multi-step planning and iterative debugging. Such tasks may necessitate multiple attempts, and longer generation sequences would further increase both time and memory overhead.

  3. The methodology largely builds upon existing approaches involving multiple sampling and test-time filtering, with the primary contribution lying in the experimental analysis. The paper needs to articulate more clearly what non-obvious insights it offers beyond existing research on test-time augmentation and model compression.

Questions for authors

Please see the critical weekness.

3 comments
AuthorsOpenReview ↗

Response, part 1/2

We thank the reviewer for the positive assessment and constructive comments. We agree that repeated sampling and test-based selection are established techniques; our contribution is a deployment analysis of when a smaller resident model, verified retry, and a finite memory and latency budget are compatible. The revision adds memory-capped experiments, timings that include verification, and long-context measurements. All changes are marked in red.

Reviewer: Sampling is straightforward; the main contribution is the speedup. Would it hold on 8–16 GB consumer GPUs, given compute and caching overheads?

Response: Two mechanisms make parallel retry inexpensive, and both carry over to consumer GPUs. First, the attempts share the prompt's KV cache: at K=8K=8 the additional completion cache for Coder-7B is about 0.13–0.15 GB (Table 21), and Table 22 reports measured process memory including serving overhead. Second, the roofline analysis in Section 3 shows that batching amortizes weight reads up to a batch size B* set by the hardware's compute-to-bandwidth ratio rather than its absolute scale. The RTX 4090 has B∗≈164B^{*} \approx 164 for bf16 weights (about 40 for 4-bit weights), close to the A100's 153 and well above the K≤8K \leq 8 used in our main comparisons.

We also timed the full pipeline directly. For a single request on B200, Coder-7B with eight attempts is 1.82× faster than one Coder-32B attempt on HumanEval+ and 1.31× faster on MBPP+, including execution of the public tests (Appendix F.2).

To test the 8–16 GB regime, we capped the GPU memory available to vLLM on an A100 at 8, 12, and 16 GB and compared each smaller model with the largest model that fits under the same cap, with test execution included (Appendix F.6). The A100 has a compute-to-bandwidth ratio similar to consumer GPUs, although absolute times differ.

  • 16 GB: the largest model that fits is Coder-14B AWQ4. Coder-7B AWQ4 with four attempts matches it (85.8% vs. 84.9% on HumanEval+, 77.3% vs. 75.6% on MBPP+) and is 1.26× and 1.43× faster.

  • 12 GB: Coder-14B AWQ4 no longer fits, while Coder-7B AWQ4 does, and with four attempts it still exceeds the 14B one-shot accuracy.

  • 8 GB: only Coder-3B AWQ4 fits (peak 7.4 GB). With eight attempts it reaches 87.7% on HumanEval+ and 75.2% on MBPP+, above the one-shot accuracy of the 7B model that does not fit. The 64 GB Apple machines were chosen so that the large reference models could be measured on the same device; the smaller retried models need far less memory.

Reviewer: Only single-turn tasks; multi-step planning and debugging need longer sequences, increasing time and memory.

Response: We agree that multi-turn agents are outside the scope of this paper, and Section 6 states this. Single-turn tasks let us measure candidate coverage and separate generation failures from verification failures, which our analysis requires. The batching mechanism applies to agentic settings as well, but longer prompts, non-shared prefixes, and repeated verification change the budget.

We now measure the systems effect of long contexts directly (Appendix F.8), with shared prompts from 127 to 16,523 tokens. With prefix caching, 16 attempts take 1.4–2.0× the time of one for prompts of 2k tokens or more, and Coder-7B with 16 attempts stays faster than one Coder-32B attempt at those lengths. Without prefix caching, the overhead grows to 3.0–7.5× and the break-even falls to four attempts at 8k tokens and beyond. Sharing the prompt's KV cache is therefore what keeps retry inexpensive at long context. Appendix B.10 additionally compares eight short attempts with a single 16k-token reasoning trace; the better strategy depends on the task.

AuthorsOpenReview ↗

Response, part 2/2

Reviewer: The paper should articulate what non-obvious insights it offers beyond existing work.

Response: The abstract and Section 1 now frame the contribution around three conditions whose interaction determines whether retry pays: correct candidates must remain reachable within a finite budget, the verifier must recognize them at acceptable cost, and recovery must fit within the measured memory and latency budget. We do not propose a new selector. The findings we consider non-obvious are:

  1. Wall time is not FLOPs. Coder-7B with eight attempts matches or exceeds the one-shot accuracy of Coder-32B on all four code benchmarks while using 4.3× less model memory and generating 1.1–4.0× faster (Table 1), because the attempts share weight reads. With the public tests included, a single request remains 1.3–1.8× faster (Appendix F.2).

  2. Fewer bits and fewer weights are different interventions. One-bit quantization lowers one-shot accuracy by up to 8.3 points but coverage by at most 1.5, so two to four attempts recover the bf16 parent (Table 2). Shrinking the model instead reduces coverage itself, by up to 13.1 points on LiveCodeBench, and requires deeper retry.

  3. Verification fails in distinguishable ways. Some tasks have no correct candidate even after deep sampling, while on others the generated tests give correct and incorrect candidates identical outcomes. Our diagnostic separates these failure modes (Section 5) and shows, for example, that BigCodeBench is limited mainly by test discrimination rather than by candidate generation.

  4. Memory savings do not imply speed. Under wide parallel decoding, Coder-7B AWQ4 at K=8 runs at 0.67–0.69× the speed of its bf16 version (Table 18, Appendix B.12), and the same one-bit deployment is slower than its bf16 parent on M3 Max but faster on M5 Pro (Table 7).

Together, these findings give practitioners a concrete criterion, Krec≤KsysK_{rec} \leq K_{sys}, for deciding when a smaller model with verified retry should replace a larger one, and explain both the favorable and the unfavorable operating points we observe.

Thank you for the author's response. I carefully reviewed the discussion of other reviewers and the author's reply. The author has addressed some of my concerns. I have decided to maintain my score.

Reviewer_wpfJOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

he paper studies the effect of retries on a smaller or more compressed model and selecting among its outputs using public or model-generated tests at a test-time compute against large model one-shot accuracy.

Critical strengths

The paper demonstrates both the cases: where the retires helped and where it doesn’t as this is not an one solution “fits” all idea. They varied the empirical analysis across model sizes, quantization methods, hardware.

Critical weaknesses

Major Weakness

  1. In both the public tests and model-generated tests, Verification costs (includes generation cost in model-generated tests) is excluded from the retry cost.
  2. K, in the paper, is often chosen by looking at held-out accuracy and comparing it to the reference one-shot accuracy, but a real deployment couldn’t choose it in advance. It is unclear and without a practical way to choose it for general task beyond these saturated benchmarks.
  3. The paper targets single-turn code generation task for the local deployment but it could use the same experiments on other tasks such as conversation-like/multi-turn, writing, summarization, translation, QA/search, etc common in local deployment use cases. Similarly for a model other then Qwen models will help us generalize beyond one model family.
  4. There still might be a lot of other factors that might affect the K beyond what is considered here such as inference engine (vllm, Ollama, HF etc), prompt size, KV & prefix caching, domains/task types etc.
  5. Verifiers for coding are cheap and simple and errors are localized in single-turn code generation. The verifiers becomes complex and errors compounds in multi-turn tasks (even coding) typical in local deployment.

Minor Weakness:

  1. From the novelty perspective, it is unclear how the paper is different from [Snell et al., 2025, Kim et al., 2026; Hao et al., 2026]. Line 87 simply mentions the scale/precision vs retry tradeoff.
  2. The paper conveniently skipped the different length generations, which seems necessary when we retry to match the large model one-shot performance.
  3. Even though the author have mentioned the average number of tests produced in some of the benchmarks, no ablation on how many tests are needed in average, which decides the verification cost.
  4. The tables could use confidence intervals to clearly see the improvements. I have a hard time trying to see where the retry helps and where it doesn’t.
  5. Ablation on the temperature and top-k/p to study their effect on candidate coverage on the public/model-generated tests is missing.
Questions for authors
  1. How is this different from Best-of-N? Is the key difference focused on the effect of test-time compute specifically best-of-N in the local deployment?
  2. I feel like there is something missing theoretically in term of a bound rather than the experimental verification that could explain the K* under different test setting. (K_rec is chosen post-benchmark so it isn’t K*)
  3. What is the effect K and the prompt size? Since the KV and attention grows with the input prompt size, this will directly affect the “”generating multiple attempts can be done almost for free” argument.
  4. How will the verification cost affect the results in the experiment section especially with the CI?
  5. How is the diversity of the model- generated test enforced?
3 comments
AuthorsOpenReview ↗

Response, part 1/3

We thank the reviewer for the detailed and constructive review. In response, we timed the full pipeline including verification, added confidence intervals and the requested ablations, and extended the evaluation to mathematical reasoning. All changes are marked in red in the revised PDF.

Reviewer: Verification costs are excluded from the retry cost.

Response: We agree that this cost should be measured, and the revision does so. Table 1 now states that its times and energies cover generation, and Appendix F.2 times the complete pipeline, generation plus execution of the public tests, on HumanEval+ and MBPP+. The tests run on the CPU in a pool of pre-started worker processes whose verdicts match the standard EvalPlus harness.

Test execution is cheap relative to generation: for a single request at K=8K=8 it adds 0.02 s on HumanEval+ and 0.05 s on MBPP+ for Coder-7B, or 3–14% of request latency across the measured models. With the tests included, Coder-7B with eight attempts is 1.82× faster than one Coder-32B attempt on HumanEval+ and 1.31× faster on MBPP+, and Coder-7B AWQ4 is 2.09× and 1.57× faster. In a batch of 60 requests, the batch must also wait for its slowest test runs (a few candidates loop until the timeout), which reduces the advantage to 1.21× on HumanEval+ and removes it on MBPP+. Section 4.2 reports both regimes.

With model-generated tests, the model must also write the tests, so CodeT trades speed for accuracy and memory: Coder-7B AWQ4 with CodeT is 1.36× faster than one 32B attempt for a single HumanEval+ request but slower on MBPP+ and in large batches. We therefore make no latency claim for model-generated verification (Section 5, Appendix F.2).

Reviewer: KK is chosen by looking at held-out accuracy.

Response: The headline comparisons fix KK in advance: Table 1 uses K=8K=8 for every model and benchmark, and the one-bit Apple comparison uses K=4K=4. KrecK_{rec}, the smallest KK at which the smaller model reaches the reference, is an explanatory quantity read off the evaluation curves (Section 4.1), not a deployment rule.

For deployment, KK can be chosen without labels as the largest budget within the measured wall-time break-even on the target device. We evaluated this rule on 200 random calibration/held-out splits (Appendix F.7). For Coder-7B AWQ4 against Coder-14B AWQ4 and for Coder-7B against Coder-32B, the held-out accuracy matches or exceeds the reference in 95–100% of splits. Choosing instead the smallest K that reaches the reference on a small labeled calibration set is less reliable (54–63%), because it overestimates recovery on a small sample. On MATH-500, fixing K=8K=8 for 7B against 32B keeps the gap non-negative in every split.

Reviewer: Other tasks (writing, summarization, …) and other model families.

Response: We scope the deployment claim to tasks where verification is informative and affordable (Sections 1 and 6); writing and summarization lack a cheap correctness check, and we make no claim there. To test transfer beyond code, we added MATH-500 (Appendix E), where a learned verifier replaces executable tests. With the Qwen2.5-Math-PRM-7B reward model, eight attempts of the 7B model reach the 32B one-shot accuracy in 0.66× its time for a 32-problem batch, including scoring. Majority vote needs 16 attempts and confidence alone does not reach the target, which again shows that the verifier sets the recovery depth.

For other model families, the paper includes Granite-Code, Minitron, and R1-Distill (Appendix B.7). Granite-Code-3B with 50 verified attempts exceeds the one-shot accuracy of Granite-Code-34B by 8.5–19 points while storing 9.7× fewer weight bytes. Section 4.2 now points to these results.

Reviewer: Inference engine, prompt size, KV/prefix caching may affect KK.

Response: We agree that the systems budget depends on the serving stack, which is why we measure KsysK_{sys} rather than infer it. We use vLLM on B200 and a fixed llama.cpp revision on each Apple machine, and never compute a speedup across engines (Appendix D.3). The B200 timings in Tables 1 and 4 use the same prompts as the accuracy evaluation at 40–60 concurrent requests: Coder-7B at K=8K=8 generates 1.48× (MBPP+), 1.73× (HumanEval+), 3.99× (LiveCodeBench), and 1.09× (BigCodeBench) faster than one Coder-32B attempt.

To isolate prompt length and caching, we varied the shared prompt from 127 to 16,523 tokens (Appendix F.8). With prefix caching, 16 attempts take 1.4–2.0× the time of one; without it, 3.0–7.5×, and the break-even against Coder-32B falls to four attempts for prompts of 8k tokens or more. Sharing the prompt's KV cache is therefore what keeps retry inexpensive at long context, and Section 6 notes that extra attempts are cheap only while batching headroom remains.

AuthorsOpenReview ↗

Response, part 2/3

Reviewer: Verification is harder in multi-turn tasks.

Response: We agree, and Section 6 lists multi-turn agents as outside the scope of our evaluation. Section 5 studies an intermediate step toward that setting, in which the model must write its own verifier without public tests, and diagnoses where this fails. The bottleneck is not the size of the test writer: replacing the 7B test writer with the 32B model changes CodeT accuracy by at most 1.2 points on HumanEval+ and MBPP+ (Appendix C.3). Extending this analysis to interactive verification is a natural next step.

Reviewer: How does this differ from Snell, Kim, Hao, and Best-of-N?

Response: Our selector is best-of-N, and we do not claim a new sampling or ranking algorithm. We study a different question: whether the smallest practical resident deployment can recover a larger model's one-shot accuracy under simultaneous memory, latency, and verifier-cost constraints. Snell et al. analyze test-time compute in FLOPs, whereas local decoding shares weight reads across batched candidates, so eight attempts cost far less than eight single ones (Section 3, Table 14). Kim et al. study memory-oriented inference choices without executable selection; we measure how model size and precision interact with verified recovery and find that quantization and shrinking affect coverage very differently. Hao et al. build a mobile-NPU inference system; we provide a recovery criterion, Krec≤KsysK_{rec} \leq K_{sys}, and test it across NVIDIA GPUs, Apple Silicon, and memory-capped deployments. The verification diagnostic of Section 5 is also new.

Reviewer: Different generation lengths were skipped.

Response: Every wall time in the paper runs until the last candidate in the batch finishes, so long completions are included. Appendix F.3 now reports completion lengths: on HumanEval+ and MBPP+ the mean is 53–149 tokens, the 90th percentile is at most 266 tokens, and at most 0.03% of completions reach the 1,024-token limit. Latency follows the longest completion rather than the mean, at about 3.9 ms per token of the longest completion for Coder-7B and 13.3 ms for Coder-32B. Appendix B.10 compares eight short attempts with a single 16k-token reasoning trace.

Reviewer: No ablation on the number of tests.

Response: We added this ablation (Appendix F.4). With 1, 2, 4, 8, or 12 generated tests per task, Coder-7B CodeT@8 stays at 82.1–82.9% on HumanEval+ and rises from 74.4% to 76.4% on MBPP+, with most of the gain reached at four tests. A few tests carry most of the benefit because many generated tests pass every candidate (see the diversity response below). Test-writer size is studied in Appendix C.3.

Reviewer: Tables need confidence intervals.

Response: Appendix F.1 adds paired 95% task-bootstrap intervals for the main comparisons, and Table 13 reports them at K=50K=50. For Coder-7B against Coder-32B one-shot (difference in points, positive favors retry):

BenchmarkK=8K=8 (pp, 95% CI)K=50K=50 (pp, 95% CI)
HumanEval++4.4 [+1.0, +8.2]+5.8 [+1.8, +10.0]
MBPP++0.1 [-2.9, +3.1]+0.7 [-2.8, +4.2]
LiveCodeBench+0.7 [-2.5, +3.9]+6.3 [+1.8, +11.1]

Retry is significantly more accurate on HumanEval+ at both budgets and on LiveCodeBench at K=50K=50, and statistically indistinguishable from the 32B model elsewhere. The one-bit model at its recovery budget is significantly more accurate than its bf16 parent one-shot on HumanEval+ and LiveCodeBench.

Reviewer: Temperature / top-p ablation is missing.

Response: The main study uses T=0.8T=0.8 and top-p=0.95p=0.95, a standard setting for repeated sampling. Appendix F.5 adds fresh sweeps over T∈{0.4,0.8,1.0,1.2}T \in \{0.4, 0.8, 1.0, 1.2\} and top-p∈{0.8,0.95,1.0}p \in \{0.8, 0.95, 1.0\} on HumanEval+ and MBPP+, together with greedy one-shot baselines. Higher temperature lowers one-shot accuracy and raises coverage, but public-test selected accuracy at K=8K=8 moves by at most 2.2 points, and Coder-7B CodeT@8 by at most 0.7 points. The conclusion also holds against a greedy reference: greedy Coder-32B reaches 86.0% / 77.8% on HumanEval+ / MBPP+, while Coder-7B with eight verified attempts reaches 88.2% / 78.8%.

AuthorsOpenReview ↗

Response, part 3/3

Reviewer: A theoretical bound explaining the optimal KK is missing.

Response: The revision addresses both sides of the tradeoff. On the systems side, the roofline model of Section 3 predicts how KsysK_{sys} depends on model size, hardware balance, and concurrency. On the accuracy side, Appendix B.13 adds a bound. With per-attempt success probabilities ps(x)p_{s}(x) and pr(x)p_{r}(x) and an oracle verifier, KK attempts of the smaller model match the reference on task xx once K≥ρ(x)=log⁡(1−pr(x))/log⁡(1−ps(x))K \geq \rho(x) = \log(1 − p_{r}(x)) / \log(1 − p_{s}(x)), so the smaller model's coverage satisfies Os(K)≥Ar−Ex[pr(x)⋅Ind[ρ(x)>K]]O_{s}(K) \geq A_{r} − E_{x}[p_{r}(x) \cdot Ind[\rho(x) > K]]. For small probabilities ρ≈pr/ps\rho \approx p_r/p_s, and ρ grows only logarithmically as pr→1p_{r} \to 1, while tasks with ps(x)=0p_{s}(x) = 0 cannot be recovered. This explains why one-bit quantization, which preserves coverage, needs few attempts, whereas shrinking the model needs more. With an imperfect verifier, selected accuracy is at most Os(K)O_{s}(K), and the difference is the selection loss analyzed in Section 5. Because the bound is a sufficient condition and loose in aggregate, we read KrecK_{rec} from the measured curves.

Reviewer: How is diversity of generated tests enforced?

Response: We do not enforce diversity explicitly, and now measure it (Appendix F.4). Between 44% and 65% of generated tests are passed by every candidate, and in 15–28% of tasks an incorrect candidate passes exactly the same tests as the best correct one. This matches the diagnostic in Section 5: at K=500K=500, sig-oracle@KK remains 5.4–18.6 points below candidate coverage, most on BigCodeBench, so the limitation lies in the tests themselves; a 4.6× larger test writer changes it by at most 1.0 point on HumanEval+ and MBPP+. Methods that make generated tests more discriminative, such as S∗S^* and DiffCodeGen, are complementary and could be combined with retry.

We hope the new measurements and analyses address the reviewer's concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.