- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PairDistill studies whether the test-time cost of LLM-based code selection can be reduced by distilling a large LLM judge into a compact verifier. The method constructs membership and equivalence queries from sampled program candidates, obtains judgments from Qwen3-8B, checks these judgments against execution-derived outcomes, and fine-tunes a Qwen2.5-1.5B-Instruct student on the resulting supervision.
The evaluation covers HumanEval+MBPP and two TACO subsets using candidates generated by Qwen2.5-3B. At inference, the main membership-argmax strategy scores each candidate with the verifier and selects the highest-scoring one using eight queries per task. Beyond measuring selection quality and cost, the paper uses ablations to examine what drives the observed gains and whether query-level verifier metrics reliably predict downstream selection performance.
Critical strengths
1. A compact verifier with measurable selection gains
The paper demonstrates that a 1.5B verifier can improve code selection over random choice, with statistically significant gains on both clean TACO subsets and a borderline gain on HumanEval+MBPP. The HumanEval+MBPP result is obtained without training on those benchmarks, providing evidence that the learned selection signal transfers beyond the TACO training distribution.
2. An informative mismatch between verifier metrics and selection quality
The ablations expose a useful mismatch between query-level verifier metrics and downstream selection quality: models with comparable or even better dev accuracy/AUROC can still perform near random when selecting among candidates. This observation is important because it shows that aggregate verifier metrics are not reliable proxies for the actual code-selection objective.
Critical weaknesses
1. The contribution of the pairwise component is not isolated
Pairwise judgment distillation is central to the paper’s framing, yet the best accuracy–cost trade-off is achieved by membership-argmax, and the equivalence-based inference strategies do not outperform it on the combined HumanEval+MBPP or clean TACO evaluations. More importantly, the paper does not compare joint membership/equivalence training against a membership-only verifier under the same training budget and evaluation protocol, so it remains unclear what pairwise supervision contributes beyond a simpler pointwise verifier.
2. The experiments do not adequately support the paper’s main mechanistic attribution
The 2×2 study shows that the original filtered-vs.-raw comparison was confounded by class balance, but it does not establish that class prior is the factor responsible for the end-to-end gains: the direct effects of both filtering and balancing are statistically nonsignificant, while the significant filtered+balanced versus raw+natural comparison changes both factors simultaneously. In addition, the two natural-prior conditions use different positive rates (7.3% vs. 13.3%), so the evidence supports that class balance may matter, but not the stronger claim that “the execution-filtering step itself is not responsible for end-to-end gains.”
Questions for authors
-
Can the authors provide a membership-only training baseline under the same student architecture, training budget, and membership-argmax evaluation? This comparison would clarify whether equivalence-query supervision contributes to the reported gains beyond a simpler pointwise membership verifier.
-
What evidence supports attributing the end-to-end gains specifically to class balance rather than execution filtering? In the reported 2×2 study, the direct filtering and balancing comparisons are statistically nonsignificant, while the significant filtered+balanced versus raw+natural comparison changes both factors simultaneously.