- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper asks when a smaller/compressed code model can replace a single attempt from a larger model under a memory budget. The smaller/compressed model runs K times in parallel, and a verifier selects one output. The motivation is that batch-size-1 decoding is bandwidth-bound, so extra parallel candidates cost little until the batch reaches the bandwidth upper bound.
The experiments use two model sets:
Qwen2.5-Coder at 1.5B–32B, in bf16, 8-bit, and 4-bit.
A controlled bf16 / 4-bit / 1-bit triplet of Qwen3.6-27B.
Paper tests on HumanEval+, MBPP+, LiveCodeBench, and BigCodeBench.
The paper explains that compression lowers pass@1 much more than oracle@K. Under self-verification, the paper splits failures into three sources: candidate generation, test discrimination, and aggregation. Apple Silicon measurements show that the upper bound K depends on the device.
Critical strengths
-
A practical, well-posed question. Framing the trade-off as K_rec ≤ K_sys is a useful lens for deployment.
-
The reliability/reachability distinction is supported. From 3B to 32B, pass@1 rises 7.7–16.3 pp on three benchmarks, while oracle@50 varies by only 1.3–2.4 pp.
Critical weaknesses
- There appear to be internal inconsistencies.
Table 1 vs. Table 21. The two tables give different values for the same configurations: Coder-7B K=8 on MBPP+: 77.8 vs. 78.0. Coder-7B K=8 on LiveCodeBench: 27.5 vs. 27.7. Coder-32B K=1 on HumanEval+: 85.1 vs. 85.3.
CodeT exceeds the oracle. In Table 5 (LiveCodeBench), CodeT@8 is 37.2 / 37.1 / 38.1%, which is higher than oracle@50 (35.4 / 34.3 / 34.3%) for all three checkpoints.
-
All evidence comes from code generation. The title and abstract present a general trade-off, but all four benchmarks are code generation.
-
Positioning relative to prior test-time scaling work could be sharpened. Several of the paper’s central observations are consistent with, and in part anticipated by, earlier work[1][2][3][4].
[1] Large Language Monkeys: Scaling Inference Compute with Repeated Sampling.
[2] Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Parameters for Reasoning.
[3] Not All Bits Are Equal: Scale-Dependent Memory Optimization Strategies for Reasoning Models.
[4] The Limits of Inference Scaling Through Resampling.
Questions for authors
-
How can CodeT@8 exceed oracle@50 on LiveCodeBench in Table 5?
-
Do the reliability-vs-reachability pattern and the K_rec ≤ K_sys hold on a math benchmark or other benchmarks?
2 comments
Response, part 2/2
Reviewer: Positioning relative to prior test-time scaling work [1]–[4] could be sharpened.
Response: We revised Section 2 and the contributions. Our selector is standard best-of-N. The question we study is different: whether a smaller or compressed model that fits in memory can replace a larger one under simultaneous memory, accuracy, and wall-time constraints, including the verification that makes the extra samples usable.
[1] and [2] budget test-time compute in FLOPs or dollars. In local decoding the cost is dominated by reading the weights, which parallel attempts share, so eight 7B attempts on MBPP+ take only 1.62× the generation time of one at 60 concurrent requests (1.73× for a single request), not 8×. Section 3 explains this gap between FLOP cost and wall time, and our measurements quantify it. The success condition in [2] corresponds to what we measure as finite-budget candidate coverage.
[3] studies memory-constrained inference choices without executable verification. With public tests, we find that even a one-bit 27B model with 3.8 GB of weights recovers the one-shot accuracy of its 14× larger bf16 parent within two to four attempts (Table 2), because quantization lowers reliability while leaving coverage nearly intact.
[4] shows that imperfect verifiers limit resampling. We characterize when this happens: public-test selection lets Coder-7B reach or exceed the Coder-32B one-shot accuracy on all four code benchmarks, whereas generated tests give an incorrect candidate the same outcomes as the best correct one on 15–28% of tasks (Appendix F.4). Our diagnostic in Section 5 separates candidate-generation, test-discrimination, and aggregation failures.
The revised contributions (Section 1) are organized around (i) the different effects of shrinking and quantization on one-shot reliability versus coverage; (ii) a measured, verifier-inclusive break-even in place of a FLOP-only model; and (iii) the separation of generation, test-discrimination, and aggregation failures. Coder-7B at K=8 generates 1.1–4.0× faster than Coder-32B one-shot and uses 1.0–3.8× less GPU energy (Table 1), and remains 1.3–1.8× faster for a single request once the public tests are run (Appendix F.2).
Response, part 1/2
We thank the reviewer for the careful reading and constructive feedback. Both inconsistencies are resolved in the revision, and we have added a transfer study on mathematical reasoning and a sharper comparison with prior work. All changes are marked in red in the revised PDF.
Response: Thank you for catching this. Accuracies at intermediate K are Monte Carlo estimates over random K-subsets of each fixed candidate pool, and the two tables had been filled from different estimation runs. In the revision, Tables 1 and 4 use a single set of estimates, with Coder-7B at K=8 reaching 78.0% on MBPP+ and 27.7% on LiveCodeBench. Table 21 now reports accuracy only at K=1 and K=50, where it is computed exactly from the full pool and matches pass@1 and exec@50 in Table 9; Coder-32B on HumanEval+ is therefore 85.1% at K=1. Intermediate budgets are shown in Table 1 and Figure 1. Section 4.1 now describes the procedure: K=1 and K=50 use the full pool, and intermediate K averages 120–300 random subsets per task (standard error at most 0.15 points).
The changes are at most 0.3 points and leave every conclusion unchanged. Appendix F.1 adds paired task-bootstrap intervals for the main comparisons: at K=8, Coder-7B is significantly more accurate than Coder-32B one-shot on HumanEval+ (+4.4 [+1.0, +8.2] points) and statistically indistinguishable from it on MBPP+ (+0.1 [−2.9, +3.1]) and LiveCodeBench.
Response: The reviewer is right that this cannot happen on a common candidate pool and task set. The printed CodeT values were computed on the 82 LiveCodeBench tasks for which all three checkpoints produced usable generated-test suites, while oracle@50 was computed on all 175 tasks, so the two columns were not comparable. We have removed these CodeT cells from Table 5.
To provide the comparison on a common basis, we generated fresh candidates for all 175 tasks with each checkpoint and computed CodeT and the oracle on the same candidates (Appendix F.9). CodeT@8 is 16.0%, 18.2%, and 19.1% for the one-bit, AWQ4, and bf16 checkpoints, against oracle@8 of 26.3%, 30.4%, and 30.8%. These values are now reported in the Table 5 caption and Appendix B.1: on LiveCodeBench, model-generated tests recover much less than public-test selection.
The public-test results for the one-bit model in Table 2 are unaffected: they use the full 175-task pool, where exec@50 = 34.9% is below oracle@50 = 35.4%. We have also clarified in Section 5 that the signature-group statistic sig-oracle@K is a diagnostic of how much the test outcomes can resolve, while candidate coverage (oracle@K) remains the upper bound.
Response: We added a transfer study on MATH-500 (Appendix E) with Qwen2.5-Instruct 3B, 7B, and 32B in bf16 and the AWQ4 builds of 7B and 32B. Math has no executable tests, so we compare confidence, majority vote, and the Qwen2.5-Math-PRM-7B reward model, with coverage as the upper bound.
Both findings carry over. Shrinking the model lowers reliability but not reachability: pass@1 rises from 65.0% (3B) to 82.6% (32B), while coverage at K=100 is 96.0%, 96.6%, and 97.0%. Quantization behaves the same way: 32B AWQ4 recovers the one-shot accuracy of its bf16 parent within two majority-voted attempts.
The verifier sets the recovery depth. For 7B → 32B, Krec is 3 under the oracle, 8 with the reward model (bootstrap interval [4, 8]), and 16 with majority vote, while confidence alone never reaches the target. Measured end to end with both models resident and scoring included, eight 7B attempts take 0.66× the time of one 32B attempt for a 32-problem batch and 0.36× for a single request, using 39–64 GB of device memory against 77–82 GB for the 32B model.
The criterion Krec≤Ksys also identifies where retry does not pay: for 3B → 7B, majority vote needs 16 attempts against a measured systems budget of 4, and for 3B → 32B the reward model needs 32 attempts against a budget of 16. We have accordingly scoped the main deployment claim to tasks with informative and affordable verification (Sections 1 and 6).