acceptodds
Under review as a conference paper at ICLR 2027

PairDistill: Amortizing Test-Time Pairwise LLM Oracles for Code Selection via Pairwise Judgment Distillation

Abstract

Pairwise LLM oracles can select the best candidate from a set of sampled programs: ExPairT-LLM poses membership and equivalence queries to a frontier LLM at test time and attains strong pass@1, but pays the cost of dozens of frontier-model queries per task at inference time. We conduct a critical analysis of distilling such pairwise oracles into compact trained verifiers. As the analysis vehicle, we build PairDistill, which amortizes this oracle into a compact trained verifier: a frontier teacher (Qwen3-8B) annotates pairwise queries once during training; because code differentiating inputs are executable, we score teacher labels against ground-truth execution outcomes before distillation (execution-scored distillation); a 1.5B student is then trained on the teacher's judgments with swap augmentation and answer-token loss masking. At inference the verifier replaces teacher generation with a single forward pass per query (a 5.3 smaller model, no decoding), using 8 verifier queries per task. On HumanEval+MBPP and TACO with Qwen2.5-3B candidates, the 1.5B verifier improves pass@1 over random selection by (HumanEval+MBPP, borderline), (TACO-fn) and (TACO-stdin) points, recovering 13–15% of the oracle–random gap (23% with a 3B verifier) at 14% of the teacher's per-task query count (8 vs. 56 queries; under 8% in token terms, since each student query is a single 1.5B forward pass over 4.1K tokens while each teacher query requires 8B generation of a differentiating input and verdict), and exceeds the teacher's own test-time selection quality (64.8% teacher vs. 67.7% student on the full HumanEval+MBPP evaluation; paired studentteacher, CI ). Gains over random are statistically significant on both TACO benchmarks (paired bootstrap, 95% CI). Ablations show that distillation volume matters far less than dev metrics suggest (25% of data recovers most selection quality), that verifier scale exhibits a threshold effect (0.5B verifiers match random selection despite higher dev accuracy than 1.5B), that task-level leakage between teacher annotation and evaluation inflates apparent selection gains by several points, and — via a de-confound — that the execution-filtering step itself is not responsible for end-to-end gains: under a unified training schedule, the class prior (not label filtering) is the largest axis of variation, and the headline configuration (filtered + balanced) is the only cell whose selection gain over random is individually significant, so the distillation recipe with a balanced query prior carries the result.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
code generation, pairwise verification, knowledge distillation, LLM oracles, program selection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TbkoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PairDistill studies whether the test-time cost of LLM-based code selection can be reduced by distilling a large LLM judge into a compact verifier. The method constructs membership and equivalence queries from sampled program candidates, obtains judgments from Qwen3-8B, checks these judgments against execution-derived outcomes, and fine-tunes a Qwen2.5-1.5B-Instruct student on the resulting supervision.

The evaluation covers HumanEval+MBPP and two TACO subsets using candidates generated by Qwen2.5-3B. At inference, the main membership-argmax strategy scores each candidate with the verifier and selects the highest-scoring one using eight queries per task. Beyond measuring selection quality and cost, the paper uses ablations to examine what drives the observed gains and whether query-level verifier metrics reliably predict downstream selection performance.

Critical strengths

1. A compact verifier with measurable selection gains

The paper demonstrates that a 1.5B verifier can improve code selection over random choice, with statistically significant gains on both clean TACO subsets and a borderline gain on HumanEval+MBPP. The HumanEval+MBPP result is obtained without training on those benchmarks, providing evidence that the learned selection signal transfers beyond the TACO training distribution.

2. An informative mismatch between verifier metrics and selection quality

The ablations expose a useful mismatch between query-level verifier metrics and downstream selection quality: models with comparable or even better dev accuracy/AUROC can still perform near random when selecting among candidates. This observation is important because it shows that aggregate verifier metrics are not reliable proxies for the actual code-selection objective.

Critical weaknesses

1. The contribution of the pairwise component is not isolated

Pairwise judgment distillation is central to the paper’s framing, yet the best accuracy–cost trade-off is achieved by membership-argmax, and the equivalence-based inference strategies do not outperform it on the combined HumanEval+MBPP or clean TACO evaluations. More importantly, the paper does not compare joint membership/equivalence training against a membership-only verifier under the same training budget and evaluation protocol, so it remains unclear what pairwise supervision contributes beyond a simpler pointwise verifier.

2. The experiments do not adequately support the paper’s main mechanistic attribution

The 2×2 study shows that the original filtered-vs.-raw comparison was confounded by class balance, but it does not establish that class prior is the factor responsible for the end-to-end gains: the direct effects of both filtering and balancing are statistically nonsignificant, while the significant filtered+balanced versus raw+natural comparison changes both factors simultaneously. In addition, the two natural-prior conditions use different positive rates (7.3% vs. 13.3%), so the evidence supports that class balance may matter, but not the stronger claim that “the execution-filtering step itself is not responsible for end-to-end gains.”

Questions for authors
  1. Can the authors provide a membership-only training baseline under the same student architecture, training budget, and membership-argmax evaluation? This comparison would clarify whether equivalence-query supervision contributes to the reported gains beyond a simpler pointwise membership verifier.

  2. What evidence supports attributing the end-to-end gains specifically to class balance rather than execution filtering? In the reported 2×2 study, the direct filtering and balancing comparisons are statistically nonsignificant, while the significant filtered+balanced versus raw+natural comparison changes both factors simultaneously.

Reviewer_idyrOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes PairDistill, which distills membership and equivalence judgments from a large language model into a smaller verifier to reduce the inference cost of code selection. Experiments on HumanEval, MBPP, and TACO show that the method achieves better pass@1 than random selection with fewer verifier queries, while reaching performance comparable to the teacher model. The authors also conduct several ablation studies to examine how class balancing, model size, and training data volume affect selection performance.

Critical strengths

S1: The paper addresses a practical problem. By moving expensive LLM judgments from inference time to training time, the proposed method offers a way to reduce the cost of selecting code candidates.

S2: The paper includes several useful ablation studies. In particular, the 2×2 experiment separates the effects of label filtering and class prior, and the results show that query-level verifier metrics do not always reflect actual code selection performance.

Critical weaknesses

W1: The paper lacks an important baseline that directly trains the verifier using execution ground-truth labels. Since the authors already have ground-truth labels for all training queries, it is unclear whether distilling teacher judgments provides any benefit over direct supervision.

W2: The paper lacks a clear overview figure of the proposed method, making the overall pipeline harder to follow. Some references are also missing, such as the Qwen3 technical report, even though Qwen3 is used as the teacher model.

W3: The paper lacks a direct comparison with the original ExPairT-LLM under the same experimental setting. The current teacher baseline uses membership-argmax rather than the full ExPairT-LLM selection protocol, making it difficult to assess how much of the original method's selection performance is preserved.

W4: The paper does not sufficiently demonstrate the necessity of pairwise equivalence distillation. The best-performing membership-argmax strategy only uses individual candidate correctness scores at inference time, but the paper does not include an ablation study that trains the verifier using only membership queries.

Questions for authors

Q1: Could the authors add a baseline that directly trains the verifier using execution ground-truth labels, and compare it with teacher judgment distillation under the same training data size and compute budget?

Q2: Could the authors add an overview figure to better explain the training and inference pipelines, and check and complete the missing references, especially the Qwen3 technical report?

Q3: Could the authors evaluate the original ExPairT-LLM selection protocol on the same candidate pools and directly compare its pass@1 and inference cost with PairDistill?

Q4: Could the authors add a membership-only training ablation to show whether equivalence queries actually improve the final code selection performance?

Reviewer_6DatOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PairDistill trains a small code verifier on judgments collected from a Qwen3-8B teacher, using execution outcomes to score or filter labels. A balanced 2,752-query subset trains a 1.5B student with answer-token masking and optional swap augmentation. Among eight code candidates, pointwise membership-argmax is the strongest inexpensive selection strategy. The paper reports gains over random selection of 3.2 points on HumanEval+MBPP, 2.4 on TACO-fn and 1.4 on TACO-stdin. Its main analytical observations are that query-level accuracy/AUROC need not predict selection, a 0.5B verifier provides little gain, and class balance appears more important than execution filtering in a limited factorial ablation.

Critical strengths

The evaluation includes useful negative and diagnostic results. The paper does not hide that teacher filtering has no statistically resolved selection benefit in its 2×2 study, or that a smaller verifier can have good dev metrics without selecting better programs. Comparing several selection strategies on fixed candidate pools exposes a meaningful distinction between query prediction and within-pool ranking (Tables 1–4).

The reported transfer result and cheap inference setting are practically interesting. Training uses TACO while HumanEval/MBPP are reserved for selection evaluation. The same-model untrained and generation-likelihood baselines help establish that the learned score has some utility. Eight small-model scoring passes are an attractive operating point, even though the exact cost advantage still needs measurement (Sections 3–5).

Critical weaknesses

The evidence does not isolate the value of pairwise oracle distillation over simpler execution-supervised membership training. Ground-truth outcomes are already computed for every training query, and the best inference strategy uses only pointwise membership scores. The paper lacks matched membership-only and direct-execution-label training controls, so it is unclear what the pairwise teacher contributes beyond standard correctness supervision and class rebalancing. Moreover, the cited ExPairT-LLM uses pairwise membership to compare output sets, whereas this paper defines membership as whether one program passes tests. This protocol change needs an explicit mapping before framing the result as amortizing that oracle or comparing against its roughly 56-query cost (Sections 3.1–3.5, 5.3).

Several inferential and efficiency claims exceed the reported evidence. The main HumanEval+MBPP improvement is borderline, and the student–teacher interval includes zero. Pooling 1,182 task×seed observations for a McNemar test reuses the same 591 tasks and does not establish independent pairs; task dependence should be retained. In the factorial study, individual balancing and filtering contrasts are both non-significant, so a larger point estimate is insufficient to establish which mechanism carries the result. Finally, the query/token ratios and amortization threshold mix changes in model, query semantics and selection algorithm, without a measured equal-protocol latency or compute comparison (Sections 5.1–5.3).

The contribution is a recombination of mature components, and the specific problem it solves is underspecified. Pairwise-judgment distillation (JudgeLM, PairJudge RM, Prometheus 2), candidate selection among sampled programs (CodeT, B4, ExPairT-LLM) and outcome-labeled verifier training (cf. CodeRM-8B) are each established techniques; the only new step is their particular composition, and the manuscript never crisply states which failure mode of prior work this combination removes — the per-task inference cost of frontier oracles, the unreliability of generation-likelihood selection, or the brittleness of execution-based selection without trustworthy tests. Moreover, the pairwise element is never isolated: the strongest inference strategy is pointwise membership-argmax, ground-truth execution labels are already computed for every training query, and the 2×2 de-confound attributes the end-to-end gain to the class prior rather than to the pairwise teacher. A matched membership-only / direct-execution-label control is required before the pairwise mechanism can be counted as a novel contribution.

Questions for authors
  1. Under the same task split, budget and schedule, what happens with membership-only training and with direct execution labels on all queries, including teacher-disagreement cases? These controls would determine whether pairwise teacher supervision is necessary.

  2. Please map the membership/equivalence queries and test-time input requirements to the original ExPairT-LLM protocol, and state whether any test-time differentiating inputs use hidden reference tests.

  3. Please recompute the multi-seed uncertainty by resampling tasks jointly across seeds or using a suitable hierarchical analysis. Does the claim of significance survive? Please also give seed variability for the four factorial cells before attributing the effect primarily to the class prior.

  4. Please give measured inference time, input/output tokens and training cost for teacher and student under the same membership protocol, and derive the break-even deployment volume from those costs rather than query counts alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.