acceptodds
Under review as a conference paper at ICLR 2027

Sample Efficient Ranking of Language Models Using Pairwise Comparisons

Abstract

The frontier of large language model (LLM) research is advancing rapidly, with new models being released every few weeks. In this fast-paced environment, rigorous evaluation is crucial to accurately assess a model's quality. Yet evaluation prompts vary in how much evidence they provide about the relative quality of a model pair. In this work, we study how to reduce evaluation cost by adaptively selecting both model pairs and evaluation prompts while controlling comparison errors. Our contribution is twofold: First, we formulate pairwise LLM evaluation as a sequential hypothesis-testing problem and develop an e-process-based procedure. We further derive a sampling distribution for actively selecting evaluation prompts. Second, we construct a complete LLM ranking by using this pairwise evaluation procedure as a black-box comparison function within a sorting algorithm. To further reduce evaluation cost, we adaptively allocate the error budget across pairwise comparisons. Empirically, across both synthetic and real-world setups, we observe that our approach can efficiently rank LLMs using only 7-22% of the total evaluation budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Evaluation, Sample Efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7Pg2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper ranks LLMs from pairwise human preference labels while using fewer labels. Each model pair is formulated as a hypothesis testing problem with two e-processes. The evaluation stops when one e-process reaches 1/α\alpha, so the probability of returning the worse model is at most alpha. Prompts are sampled from a distribution that favors prompts with clear annotator preference, and an LLM predicts the preference probability of each prompt. The pairwise evaluator is used as a black-box comparison function within quicksort or merge sort, and the error budget alpha is allocated across pairs by a convex program. The paper provides bounds on the expected number of labels for pairwise evaluation and full ranking. Experiments on LMArena, PPE, and UltraFeedback show that ranking with adaptive alpha uses about 7-22% of the labels, while achieving ranking quality close to Bradley-Terry ranking fitted on all labels.

Critical strengths
  1. The error guarantee does not depend on the predictor. The LLM predictor affects only the rate of evidence accumulation, while the guarantee on returning the worse model still holds (Lemma 3.1). Since the predictor is the least reliable component of the method, this property is important.

  2. The proposed error allocation reduces the number of labels. On all three datasets, adaptive alpha uses up to about 40% fewer labels than constant alpha, with similar ranking quality. Active ranking and active testing use more labels under the same setting.

Critical weaknesses
  1. My main concern is the application scenario. The paper gives no concrete deployment scenario and no online experiment. Chatbot Arena and the paper's experiments evaluated on a selected dataset are different. In practice, when would people run only a small part of a benchmark instead of the full benchmark? Lower cost is not a convincing reason. Take code testing as an example. We do not skip tests to save cost, because a missed test case may reveal a failure in some function of the code. Some methods reduce the amount of test data. However, they are proved in the code space, and the proofs do not carry over to LLM applications.

  2. Suppose the assumption above holds, that is, pruning a benchmark is valid. Then modeling the LLM evaluation process with e-processes is insightful. However, as the authors state in lines 137-143, e-process modeling is not the main contribution of the paper. Related work such as SERPANT [1] modeled the process first. But the experimental baselines do not include SERPANT.

[1] https://openreview.net/forum?id=7Y8xRnGQ47

Questions for authors

See Weakness

4 comments

We thank the reviewer for their thoughtful comments. Please find our responses below.

In practice, when would people run only a small part of a benchmark instead of the full benchmark?

Our method is tailored towards pairwise human evaluation. In this setting, cheap verification such as unit tests are either unavailable or cannot tell responses apart: two code snippets may pass the same tests even though one is cleaner and handles more edge cases. Each time a new model is released, it must be compared against existing models, and every comparison requires fresh human judgments. Examples include internal model-release evaluations on fixed datasets and online arena-style leaderboards. In our work, we simulate the online setup by sequentially accessing the labels in the LMArena and PPE datasets, which are subsets from ChatBot Arena.

Why fewer human labels is acceptable unlike skipping unit tests? Software testing aims to find any failure, so a skipped test can hide a bug in a single trajectory and cannot be extended to quantify a model’s quality. Pairwise ranking instead estimates a population preference (the probability that humans prefer one model over another). Our e-process test stops once enough evidence has been collected (irrespective of the prompts used), and the probability of declaring the wrong winner is bounded by α\alpha.

Comparison with SERPANT

SERPANT is not directly comparable to our method: it returns a partial ordering of models, not a full ranking. More importantly it has no stopping criterion. It keeps a set of unresolved model pairs and, in each round, labels randomly selected examples across all unresolved pairs, updating each pair's e-process.

For a fair comparison, we ran SERPANT with the same sample budget our method (QSA) needed to produce a full ranking, and report the fraction of model pairs it resolves:

DatasetSample budget (from QSA)Pairs resolved by SERPANT
LMArena9.7%0.0% ± 0.0 (0 pairs)
PPE21.8%0.6% ± 0.3 (≈1 pair)
UltraFeedback6.8%36.6% ± 1.1 (≈nn pairs)

Note that using the same budget that we used for our method, SERPANT resolves almost no pairs on LMArena and PPE, and only about a third on UltraFeedback. It spreads its labels uniformly over all O(n2)O( n^2 ) model pairs, so evidence builds up slowly on each one. Our method instead resolves the O(nlog⁡n)O( n \log n ) comparisons the sorting algorithm requests, concentrates labels on one pair at a time, and selects informative prompts. So while our method can return a full ranking with this budget, Serpent cannot even produce a partial ranking to compare it with.

Thank you for taking the time to review our responses. Since the discussion period ends in a few days, we wanted to double-check whether your questions have been addressed. Please let us know if you need any further clarification.

Thank you for the clarification. I tend to keep my current rating.

Thank you for acknowledging our response. We hope the additional clarification about the problem statement and SERPANT experiments addressed your concerns.

Since we are entering the final day of the rebuttal, we wished to double check if you had any remaining concerns. If your concerns have been addressed, please consider reassessing the overall evaluation accordingly. If not, we are open to any suggestions to help improve the paper in the future.

Reviewer_MCZEOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to rank language models using fewer pairwise preference judgments. It combines an e-process-based sequential comparison procedure with active prompt sampling, uses this procedure within sorting algorithms, and adaptively allocates error tolerances across model pairs. It provides theoretical analysis and experiments on synthetic data and three preference datasets.

Critical strengths

The theoretical derivations are relatively clear. The paper proposes a method for efficiently selecting evaluation prompts and ranking language models, supported by substantial theoretical analysis.

Critical weaknesses

1. The connection between the efficiency assumptions and the actual predictor is unclear.

Some of the efficiency analysis for LLM-guided sampling relies on the strong assumption sup⁡x∣π^t−1(x)−π(x)∣≤ϵ\sup_x |\widehat{\pi}_{t-1}(x)-\pi(x)|\leq\epsilon. This error bound, required for every prompt at every round, may not hold for actual LLMs. Although the appendix also considers conditions on a shortlist, it does not establish whether the actual predictor satisfies them. I would like to see weaker assumptions or additional experiments connecting these theoretical conditions to practical performance.

2. The experiments need to better establish the method's benefits, stability, and total cost.

The baselines in Table 1 do not sufficiently distinguish gains from the LLM's predictive ability from gains due to the sampling method itself. The methods use different predictors, so some of the efficiency gains may come from the LLM. I suggest comparing different sampling methods using predictions from the same LLM.

The stability of the results in Table 2 needs further validation. Although standard errors are reported, the experiments were repeated only three times. This provides limited support for small differences in ranking quality and for the claim that quality is largely preserved.

The prompt template does not specify the contents of {History} and {Questions}, and the call frequency, history truncation, and reuse of candidate prompts are unclear. Repeatedly providing the history and many questions could incur substantial token costs. Improved sample efficiency therefore does not necessarily imply lower total cost. I would like to see the number of calls, token usage, and monetary cost reported alongside the evaluation and annotation costs saved.

The paper includes some controlled comparisons and ablations, including Table 4 in the appendix, but mainly reports results at a few fixed settings. This is not sufficient to establish an efficiency advantage at comparable quality. I suggest comparing sample usage at similar accuracy or ranking quality, or comparing quality under the same sample budget, and clarifying whether the error budgets are comparable across methods. Also, Table 1's win rate against uniform sampling does not directly show the average number of samples saved; actual sample usage and uncertainty intervals should also be reported.

Questions for authors
  1. Clarification: Which efficiency guarantees apply to the practical implementation, and what evidence supports the assumptions on its predictor and shortlist selection?
  2. Clarification: How are {History} and {Questions} constructed, how often is the selector called, and how does its cost compare with the evaluation and annotation costs saved?
  3. Clarification: What error budgets were used for the fixed and adaptive methods, and are they comparable?
  4. Additional evidence: Can you compare different sampling rules using the same LLM predictions, and report sample usage at comparable accuracy or ranking quality, with uncertainty intervals?
8 comments

We thank the reviewer for their thoughtful comments and detailed suggestions. Please find our responses below.

I would like to see weaker assumptions or additional experiments connecting these theoretical conditions to practical performance.

We can relax this assumption with minor changes. In fact, we can replace the assumption sup⁡x∣π^(x)−π(x)∣≤ϵ\sup_x |\hat \pi(x) - \pi(x)| \leq \epsilon with assuming a bounded population variance: Var(π^t(X)−π(X))<ϵVar(\hat \pi_t(X) - \pi(X))<\epsilon and the exact same result holds true.

The main modification is in Lemma A.4: instead of bounding the range of d(X)=±ℓ(π^(X)−π(X))d(X) = \pm \ell(\widehat{\pi}(X) - \pi(X)) uniformly across prompts, we use the exact identity DKL(q^s∥qs)=ℓ2∫01u Var⁡qus(π^(X)−π(X)) duD_{\mathrm{KL}}(\widehat{q}^s \Vert q^s) = \ell^2 \int_0^1 u\ \operatorname{Var}_{q_u^s}(\widehat{\pi}(X) - \pi(X))\ du along the path qus(x)∝qs(x)eud(x)q_u^s(x) \propto q^s(x)e^{u d(x)}. Because π^(x),π(x)∈[0,1]\widehat{\pi}(x), \pi(x) \in [0, 1] implies qus(x)/p(x)≤eℓq_u^s(x)/p(x) \le e^\ell, we get Var⁡qus≤eℓVar⁡p\operatorname{Var}_{q_u^s} \le e^\ell \operatorname{Var}_p, which bounds both the KL loss (≤12eℓℓ2σ2\le \frac{1}{2}e^\ell \ell^2 \sigma^2) and the label-mean shift (≤12eℓ/2ℓσ\le \frac{1}{2}e^{\ell/2}\ell\sigma) in terms of σ2\sigma^2 rather than sup⁡x∣π^(x)−π(x)∣\sup_x |\widehat{\pi}(x) - \pi(x)| (so constant calibration shifts cancel and outlier prompt errors are weighted by p(x)p(x)).

Using this lemma, we updated Corollary A.3 (full-pool predictors) and Proposition A.2 (shortlisted predictors) to use variance bounds rather than sup⁡x\sup_x bounds, relaxed the every-round requirement to allow bad warmup/outlier rounds with geometrically decaying probability, and extended Proposition A.2 to our practical learned bet νt\nu_t (Eq. 12), which Corollary B.1 then plugs directly into the full ranking guarantee (Theorem 3.2).

Additional evidence: Can you compare different sampling rules using the same LLM predictions?

We conduct additional experiments to separate gains due to the LLM's predictions from gains due to our sampling distribution, we equip Active Testing's proposal distribution with the same LLM predictions (BLR-based baselines already use our sampling distribution, so they are not comparable here).

As shown below, Active Testing with LLM predictions is substantially less sample-efficient than our method on LMArena and PPE, at comparable accuracy. On UltraFeedback, it is more sample-efficient but loses 2% in accuracy. Since both methods use identical LLM predictions, the efficiency gains come from our proposed sampling distribution, not the LLM alone.

DatasetMethodWin rate (↑)Accuracy (↑)Label ratio (↓)
LMArenaLLM + Active Testing35.37%92.68%1.213
LLM + our proposal (Table 1)57.32%91.71%0.966
PPELLM + Active Testing41.03%85.87%1.057
LLM + our proposal (Table 1)56.79%85.65%0.971
UltraFeedbackLLM + Active Testing67.80%95.42%0.913
LLM + our proposal (Table 1)59.32%97.46%0.979

The stability of the results in Table 2 needs further validation. Although standard errors are reported, the experiments were repeated only three times.

We have conducted a fresh set of 5 ranking experiments and report the results below. The overall trends remain consistent with quick and merge sort with adaptive α\alpha achieving the best performances.

MethodLMArena (Acc.)LMArena (Samples %)PPE (Acc.)PPE (Samples %)UltraFeedback (Acc.)UltraFeedback (Samples %)
QS, constant α61.5 ± 0.315.0% ± 0.558.4 ± 0.331.5% ± 1.374.9 ± 0.112.1% ± 0.6
QS, doubling α61.5 ± 0.115.4% ± 0.958.5 ± 0.232.4% ± 1.175.0 ± 0.114.3% ± 0.9
QSA (ours)61.2 ± 0.210.5% ± 0.759.1 ± 0.322.2% ± 0.874.7 ± 0.17.3% ± 0.5
MS, constant α61.2 ± 0.314.7% ± 0.459.1 ± 0.128.6% ± 0.874.9 ± 0.111.1% ± 0.8
MSA (ours)60.7 ± 0.510.0% ± 0.558.6 ± 0.321.3% ± 0.874.4 ± 0.27.0% ± 0.7

Clarification: How are {History} and {Questions} constructed?

We construct the history section by randomly selecting 10 questions along with the label indicating the model that won, during sequential hypothesis testing. We provide all the unlabeled questions in the Questions section. Each input question in the history and questions section had been truncated to 500 characters.

how often is the selector called, and how does its cost compare with the evaluation and annotation costs saved?

In our framework, we use a single LLM call for each human label collected, which selects the prompt for annotation. We report the detailed token and cost statistics for Gemini 3.6 Flash used in our experiments. We observe that LLM cost to save a human label is only $0.02 to 0.1, which is very small compared to human costs required to judge complex LLM trajectories.

LMArenaPPEUltraFeedback
Avg. input tokens per call~4.7k~4.9k~43k
Cost per call~$0.008~$0.008~$0.065
LLM calls per ranking~9.1k~2.6k~4.4k
LLM cost per ranking~$70~$22~$280
Human labels saved vs. uniform sampling~3,200~580~2,800
LLM cost per human label saved~$0.02~$0.04~$0.10

Also, Table 1's win rate against uniform sampling does not directly show the average number of samples saved; actual sample usage and uncertainty intervals should also be reported.

To interpret sample efficiency properly, it helps to separate standalone pairwise testing (Table 1 in this response) from full ranking (Table 2 in this response), as the two settings have different error objectives.

In table 1, we report the label ratio of LLM-guided vs. uniform sampling. Across all pairs at fixed α=0.05\alpha = 0.05, LLM-guided sampling uses 2.1–3.4% fewer labels on average. Inspecting the failure cases shows that losses occur mainly on easy pairs (\mu is close to 0 or 1), where the e-process stops after only a few examples before the LLM has accumulated enough label history to outperform random sampling.

MethodLMArenaPPEUltraFeedback
Labels ratio (LLM / Uniform)0.966 ± 0.031 (↓3.4%)0.971 ± 0.013 (↓2.9%)0.979 ± 0.074 (↓2.1%)

Importantly, this all-pair ratio understates the impact on full ranking because easy pairs stop in just a few draws: spending even 1–2 extra labels before the LLM has enough history inflates a pair's ratio while adding almost nothing to the total label count, whereas on harder pairs that require many evaluations, LLM-guided selection saves tens of labels per pair. In full ranking with adaptive αij\alpha_{ij} (where the pilot stage also provides initial label history and sorting focuses comparisons on competitive pairs), the large absolute savings on harder comparisons dominate the total budget while the small overhead on easy pairs is negligible. As a result, switching from uniform to LLM-guided sampling inside adaptive ranking reduces total label usage by 27.7% on average (16–43% across [QSA] and [MSA], saving ~580 to ~3,200 human labels per ranking).

MethodLMArena (Samples %)PPE (Samples %)Ultrafeedback (Samples %)
Quick Sort (const. α\alpha)15.6%28.4%14.5%
Quick Sort (adaptive α\alpha)13.4% (↓14.1%)27.1% (↓4.6%)11.9% (↓17.9%)
Quick Sort (adaptive α\alpha + LLM)9.7% (↓37.8%)21.8% (↓23.2%)6.8% (↓53.1%)
Merge Sort (const. α\alpha)15.2%28.5%14.6%
Merge Sort (adaptive α\alpha)12.9% (↓15.1%)25.9% (↓9.1%)11.7% (↓19.9%)
Merge Sort (adaptive α\alpha + LLM)9.7% (↓36.2%)21.7% (↓23.9%)7.6% (↓47.9%)

I suggest comparing quality under the same sample budget

We perform additional experiments to compare the ranking quality of baselines to our method at the same sample budget. In the table below, we observe that our proposed method QSA outperforms baselines on PPE and Ultrafeedback datasets, while achieving similar results on LMArena.

Note that this setup favors the baselines: the sample budget comes from our method. ATQ and ATM needed a sweep over α\alpha to match it, and AR had to be told when to stop. Without this information, the baselines run until their own stopping criteria are met and use substantially more labels (e.g., 52–73% for AR and 17–31% for ATQ/ATM; Table 2). Our method instead decides when to stop on its own.

MethodLMArena (@ 9.7%)PPE (@ 21.7%)Ultrafeedback (@ 6.8%)
Active Ranking [AR] (stopped at budget)61.8 ± 0.458.2 ± 0.559.5 ± 2.4
Active Testing w/ QS [ATQ] (α\alpha tuned to budget)60.6 ± 0.2 (10.0%)58.6 ± 0.2 (21.5%)74.1 ± 0.5 (6.9%)
Active Testing w/ MS [ATM] (α\alpha tuned to budget)60.6 ± 0.4 (9.1%)58.2 ± 0.7 (21.5%)73.0 ± 0.8 (7.2%)
Quick Sort (adaptive α\alpha) [QSA] (Ours)61.0 ± 0.3 (9.7%)59.3 ± 0.1 (21.8%)74.7 ± 0.2 (6.8%)

Clarification: What error budgets were used for the fixed and adaptive methods, and are they comparable?

We report the average error budget (α\alpha) in the constant and adaptive settings for all datasets in the table below.

MethodLMArenaPPEUltraFeedback
Constant (QS, MS)0.050.050.05
Adaptive (QSA/MSA, ours)0.2090.2410.124

We conduct additional experiments in which each constant-α\alpha sorting run uses the average α\alpha produced by the adaptive allocation. Even with α\alpha matched, our methods still use ~13% fewer labels on LMArena and ~40% fewer on UltraFeedback, while being comparable on PPE. The matched constant α is also an oracle hyperparameter choice, it is only known after running the adaptive method. Our algorithm instead sets per-pair tolerances in a principled way without tuning α\alpha using additional heuristics.

MethodLMArena (Acc.)LMArena (Samples %)PPE (Acc.)PPE (Samples %)UltraFeedback (Acc.)UltraFeedback (Samples %)
QS, constant matched α61.0 ± 0.611.1% ± 0.458.6 ± 0.322.3% ± 1.775.0 ± 0.111.9% ± 0.6
QSA (ours)61.0 ± 0.39.7% ± 0.859.3 ± 0.121.8% ± 1.374.7 ± 0.26.8% ± 0.5
MS, constant matched α61.0 ± 0.311.1% ± 0.359.1 ± 0.222.3% ± 1.975.1 ± 0.012.6% ± 0.3
MSA (ours)60.1 ± 0.59.7% ± 0.858.3 ± 0.421.7% ± 1.374.5 ± 0.37.6% ± 0.9
Reviewer_E5onOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to reduce the number of human labels needed to rank LLMs from pairwise preferences, by choosing both the model pairs and the prompts to label. Given one pair, it runs two importance weighted betting e-process, one per direction, and stops when both of them crosses the setting threshold (1/alpha). It derives the growth optimal prompt proposal, mixes the two directions by accumlated wealth, and replaces the unknown per prompt preferences with an LLM prediction. The authors also gives proof for the error control and the stop time bound. The authors perform experiments on 3 public datasets (LMArena, PPE and UltraFeedback), and report 7-22% of label used, at ranking quality close to offline Bradley Terry.

Critical strengths

The paper proposes a new method that selects model pairs and prompts both to reduce the human annotation budge for pairwise model ranking, and support it with theoretical guarantees. And the authors also perform realistic and comprehensive experiments, which shows that the proposed method needs far fewer human labels than existing baselines at comparable ranking quality. Overall I think their setting is practical and can be applied for real applications.

Critical weaknesses

The paper presents LLM-guided prompt selection as its main novelty and the source of the label savings, but the results do not support this well. Table 1 only reports how often it needs fewer labels than uniform sampling (57–59% of trials), not the size of the savings. In Table 4 (appendix), LLM-guided sampling needs more labels than uniform sampling in 4 of 9 non-adaptive settings, and the authors themselves attribute the gains to the error allocation strategy. I would like to see the size of the per-pair savings reported, and this ablation moved into the main text with the claims adjusted to match. As it stands, it is unclear how much of the savings comes from the paper's main contribution.

Questions for authors
  1. How large are the per pair label saving in Table 1? For example, the mean ratio of labels used by LLM-guided vs. uniform sampling?
2 comments

We thank the reviewer for their thoughtful comments. Please find our responses below.

How large are the per pair label savings in Table 1? For example, the mean ratio of labels used by LLM-guided vs. uniform sampling?

We report the ratio of the total labels used by LLM and uniform sampling in the table below. We observe that across all pairs LLM based sampling uses 2.1-3.4% fewer labels compared to uniform sampling on average. We also inspected the failure cases of LLM-based sampling and found that it fails at times for easy pairs, where only few examples are enough for e-processes to conclude and random sampling is better. In these settings, the history of labels is extremely small for the LLM to use that information to effectively pick examples to label.

MethodLMArenaPPEUltrafeedback
Labels ratio (LLM / uniform)0.966 ± 0.031 ( 3.4%)0.971 ± 0.013 ( 2.9%)0.979 ± 0.074 ( 2.1%)

Note that these are average savings across all pairs, and in a ranking setup these gains add up across all comparisons. This is observed in the full ranking results where LLM-based sampling provides an average additional gain of ~27.7% in sample efficiency over adaptive α\alpha setting.

MethodLMArena (Samples %)PPE (Samples %)Ultrafeedback (Samples %)
Quick Sort (const. α\alpha)15.6%28.4%14.5%
Quick Sort (adaptive α\alpha)13.4% (↓14.1%)27.1% (↓4.6%)11.9% (↓17.9%)
Quick Sort (adaptive α\alpha + LLM)9.7% (↓37.8%)21.8% (↓23.2%)6.8% (↓53.1%)
Merge Sort (const. α\alpha)15.2%28.5%14.6%
Merge Sort (adaptive α\alpha)12.9% (↓15.1%)25.9% (↓9.1%)11.7% (↓19.9%)
Merge Sort (adaptive α\alpha + LLM)9.7% (↓36.2%)21.7% (↓23.9%)7.6% (↓47.9%)

In Table 4 (appendix), LLM-guided sampling needs more labels than uniform sampling in 4 of 9 non-adaptive settings.

In Table 4, three out of four settings where random sampling outperformed LLM happened in the QS doubling baseline, and the gaps are small (within two std. errors). In all adaptive baselines, LLM-guided samples result in a significant improvement in sample efficiency (16-43% fewer samples).

We investigated the QS doubling baseline closely and found that it sets higher α\alpha for pairwise comparison as the recursion deepens compared to others. In this regime, for easy pairs little evidence needs to be collected for the e-process to conclude. LLM-guided sampling benefits from the label history it accumulates within an e-process. When α\alpha is large, an e-process stops after only a few samples, so the LLM has little history to guide its selection, and uniform sampling is just as effective. To verify this, we replayed the LLM-guided e-processes for every model pair at increasing α and compared their sample usage with uniform sampling. LLM-guided sampling uses 2–3% fewer samples at α=0.05\alpha = 0.05 on PPE and UltraFeedback, but ~5% more samples at α\alpha = 0.4 on all three datasets. Our adaptive α\alpha algorithm avoids this regime: its pilot stage provides an initial label history before LLM-guided sampling begins, which is consistent with the large gains of LLM-guided sampling in all adaptive baselines.

Thank you for taking the time to review our responses. Since the discussion period ends in a few days, we wanted to double-check whether your questions have been addressed. Please let us know if you need any further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.