acceptodds
Under review as a conference paper at ICLR 2027

The Cost of Comparing Coupled Generators

Abstract

Sharing randomness between two generators can make their outputs agree, reducing the work needed to compare them. Directly judging the coupled outputs, however, generally changes the win probability being estimated. We study how agreement can reduce evaluation cost while retaining the independent-output win probability. Let be the average probability that the coupled outputs differ. We give a procedure that does not know and, at accuracy and confidence , uses expected generation and judgment costs of order and , respectively. When , both dependences are necessary on the same worst-case instance, even for comparators induced by a strict scalar ranking and algorithms that adaptively judge outputs they have not generated. A pilot based only on output identity provides the unknown mismatch scale. The analysis connects crossed paired comparisons and reference differences, two ways to retain the target and exploit agreement. Recorded-pool experiments with language models on two text-generation tasks show 29–49% lower RMSE for crossed comparisons than independent pairing at the same generation count, with fewer judgments on all four pools.

Primary area:
learning theory
Keywords:
pairwise evaluation, coupled generation, common random numbers, win probability, language model evaluation, query complexity, lower bounds, LLM-as-a-judge

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hZveOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies how to cost-effectively evaluate the pairwise win rate between two LLM generators with provable accuracy guarantees.

Critical strengths

This work leverages the agreement from coupled generation to construct cross-comparisons, enabling unbiased estimation of the win rate for independent outputs while simultaneously cutting both generation and judgment costs.

Critical weaknesses

The zero-judgment shortcut in paper requires the two fully decoded responses to be identical, including whitespace. For long texts, high temperatures, and open-ended generation, the probability that two complete responses are exactly the same can be very low.

Questions for authors

reference Weaknesses

2 comments

Dear Reviewer hZve,

Thank you for your careful reading and for raising the role of exact matches in longer and more open-ended responses. We have clarified the mechanisms and organized the supporting evidence as follows.

  • Section 2.3 explains how agreement and score concentration affect the sampling requirement.
  • Sections 4.3–4.4 and Tables 3–4 report the longer-response and larger-temperature comparisons and isolate the contribution of cancellation.
  • Section 4.5 and Appendix F.2 add WildBench results under four judges and an ablation without cancellation.

Next, we respond to your comment.

The zero-judgment shortcut in paper requires the two fully decoded responses to be identical, including whitespace. For long texts, high temperatures, and open-ended generation, the probability that two complete responses are exactly the same can be very low.

Exact matches enable the double-match cancellation shortcut. The crossed estimator also reduces the sampling requirement through score concentration. Section 2.3 now explains these effects using Eq. (6), which separates how often a block contains a mismatch from how closely its score concentrates around half credit. Theorem 2 connects this variance to the expected stopping cost. Crossing independent coupled pairs preserves the target win probability whether or not their responses match (pp. 3–4).

The narrative task illustrates the distinction. Under the Qwen2.5-7B-Instruct judge, double matches determine 1.94% of block scores without judging, while crossed-score variance is 0.0504 compared with 0.0726 for Independent 2. Both methods use four responses per block. Holding scores and stopping times fixed, cancellation saves 0.6% of distinct judgments. This separates the judgments skipped within a block from the savings due to collecting fewer blocks. Section 4.4 and Table 4 report this analysis (pp. 8–9).

For CommonGen prompts requesting 80–100-word narratives, Crossed 2 uses 29.5% fewer generated responses and 31.2% fewer distinct judgments. For E2E at temperatures 0.70 and 1.00, the reductions are 12.5% and 19.2%. These reductions compare with the lowest independent mean for each resource at error tolerance 0.02 and 95% confidence. Section 4.3 and Table 3 report the results (pp. 7–8).

On WildBench with Mistral Small 3 24B, Crossed 2 saves both resources under Qwen, Haiku, Sonnet, and GPT, with reductions of 1.9–12.6% in responses and 14.8–20.3% in distinct judgments at the same precision. Double matches determine 2.08% of block scores. With cancellation disabled, Crossed 2 still requests 12.0–18.9% fewer comparisons than the lowest independent count. Table 5 reports distinct judgments after caching, and Table 9 reports these requests before caching. The results appear in Section 4.5 and Appendix F.2 (pp. 9 and 21–22).

Thank you for your response. After considering it, I have decided to maintain my original score.

Reviewer_TbC8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Well, this seems like another LLM-generated paper. The authors propose a so-called "cross comparison" technique to reduce the cost of binary comparison in RLHF. The authors may want to replace the traditional RMs used in RLHF.

Critical strengths

None.

Critical weaknesses
  • This paper is a little hard to understand. It does not follow the traditional paradigm of previous papers in this field: raise/discover a problem, explain how to solve the problem, and show evidence.

  • This paper does not compare with previous papers on some common benchmarks, though the authors mention some of these benchmarks in the RELATED WORK section. This makes it hard for the reviewer to judge the method proposed in this paper.

  • It would be better if the authors could provide some results on popular reward modeling benchmarks like RewardBench, etc.

Questions for authors

Please check the Critical Weaknesses.

3 comments
AuthorsOpenReview ↗

Initial Clarification about our task and method

Dear Reviewer TbC8,

Thank you for your time and effort in reviewing our paper. We are preparing new materials to address the reviewers' concerns. We respectfully believe that the main concerns come from a misunderstanding of the task, so we would like to clarify it first. A full response with additional experiments will follow.

  1. Our task is fundamentally different from "replacing the traditional RMs used in RLHF"

A reward model judges which of two responses is better, but we ask which of two models is better. In our task, given two models A and B (e.g., two checkpoints, or the same model with different decoding settings), we often use an LLM judge to decide which one is better, but this process is costly. For example, given a dataset of 10K prompts, the most straightforward way is to make 10K generation calls to model A, 10K to model B, and another 10K calls to the judge, which gives the win rate of A over B on this dataset. We ask whether A and B can be compared reliably at a much lower cost, fewer generation calls and/or judge calls.

A standard way is random sampling, draw prompts at random, generate and judge only those, and stop once the win rate can be reliably estimated, such as within ±0.02 at 95% confidence. This needs a few thousand samples regardless of the dataset size. This is the baseline we compare.

In this paper, we further reduce the cost by running A and B with shared random seeds, under which similar models often produce identical responses. We show how to exploit this without biasing the estimate and reduce both the generation calls and the judge calls. For example, comparing Qwen2.5-0.5B at two temperatures on E2E, standard random sampling needs about 11,000 generated responses and 4,200 judge calls to reach ±0.02 at 95% confidence, while our method needs 4,400 responses and 1,300 judge calls for the same win rate and precision. We got about 60% fewer generations and 70% fewer judge calls. We also prove matching lower bounds, showing these savings cannot be improved beyond constant factors.

We summarize the differences between our task and reward modeling in the following table.

Reward model (RLHF)Our paper
QuestionWhich of two responses is better?Which of two models is better?
InputA prompt and two given responsesTwo models (or configurations), a fixed judge, and a set of prompts
OutputA preference or score for each response pairThe win rate of model A over model B, with a confidence guarantee
Need Ground truth?Yes. Human or verified preference labelsNo. the target is the win rate as defined by the given judge
Role of the judgeThe object being trained and evaluatedA fixed component, an input. Any judge can be used
StageModel training, like RLHF. The RM is trained, then used to train a policyModel evaluation and selection. No training
Typical metricAgreement with labels (e.g., RewardBench)Number of generations and judge calls to reach a given precision

Because our method is not a reward model, reward-modeling benchmarks such as RewardBench do not apply to it.

  1. On the structure of the paper.

We thank the reviewer for this comment. The current version is framed from a learning-theory perspective, which may make it harder to follow for readers in the broader LLM community. The paper follows problem (the cost of pairwise evaluation, Section 1) -> solution (crossed comparison with adaptive stopping, Sections 2–3) -> evidence (guarantees in Section 3, experiments in Section 5). We are revising the framing of the paper for a general LLM audience.

We hope this clarifies the scope of the paper, and we would be glad to answer any further questions.

Dear Reviewer TbC8,

Thank you for your comments on clarity and benchmarking. We have revised the manuscript in three main ways.

  • The introduction, Figure 1, and Sections 2–3 explain the problem and construction, with a complete algorithm in Appendix A.
  • Sections 4.1–4.5 expand the comparisons across models, fine-tuning checkpoints, datasets, and judges.
  • Appendix F.3 and Table 10 add the requested RewardBench validation with three judges.

Next, we respond to each concern.

This paper is a little hard to understand. It does not follow the traditional paradigm of previous papers in this field: raise/discover a problem, explain how to solve the problem, and show evidence.

We have revised the introduction and the organization of the paper to make this reasoning explicit.

Our starting point is the cost of repeated pairwise evaluation, which requires both generated responses and judgments. Coupled generation offers an opportunity to save resources through agreement. However, directly comparing coupled responses can change the win probability being estimated. The problem we address is how to exploit this agreement while preserving the independent-output comparison target (Introduction and Section 2.1, pp. 1–2).

To address this problem, we propose crossed comparison with adaptive stopping. Crossing independent coupled pairs preserves the target and uses agreement to reduce score variance and judgment costs. Adaptive stopping then determines how much sampling is needed to reach the requested precision. Figure 1 and Sections 2–3 explain this construction (pp. 2–5).

We establish unbiasedness and the accuracy guarantee, derive generation and judgment cost bounds, and give corresponding lower bounds that characterize their dependence on mismatch probability (Sections 2–3, pp. 2–5).

Section 4 validates the approach through comparisons at matched precision across models, fine-tuning checkpoints, tasks, and judges. The results show savings in both generated responses and distinct judgments. Ablations examine where the savings come from, and online evaluation measures them during live generation and judging (Tables 1–6, pp. 6–10).

This paper does not compare with previous papers on some common benchmarks, though the authors mention some of these benchmarks in the RELATED WORK section. This makes it hard for the reviewer to judge the method proposed in this paper.

We did compare method with previous baselines. For efficiency, Independent 1 implements conventional pairwise evaluation with one independently sampled response from each generator. Independent 2 and 4 use all-pairs averaging. All methods estimate the same target at the same precision, using the same stopping rule and applicable identity, cancellation, and caching rules. Independent 2 matches Crossed 2 (our default method)'s four responses per block.

Table 1 now covers four models from 0.5B to 24B on E2E and CommonGen under Qwen and Haiku judges. Crossed 2 uses 42.8–59.2% fewer generated responses and 58.8–88.9% fewer distinct judgments than the lowest independent mean for each resource. Section 4.3 and Table 2 extend the comparison to epoch 1 and epoch 2 of one fine-tuning run at the same temperature. Section 4.5 and Table 5 add Mistral 24B on open-ended WildBench tasks under four judges, with savings in both resources under every judge(pp. 6–9).

It would be better if the authors could provide some results on popular reward modeling benchmarks like RewardBench, etc.

Our paper studies how to reduce generation and judgment costs when estimating the win probability between two generators at a prescribed precision. RewardBench assesses a judge’s agreement with preference labels on fixed response pairs, so its label-agreement metric does not directly measure these resource savings.

However, judge quality is relevant to the practical value of the resulting evaluations. To address your concern, we evaluated Qwen2.5-7B-Instruct, Claude Haiku 5.5, and Claude Sonnet 5.5 on all 2,985 pairs in the filtered RewardBench v1 split. Their agreement with the preferred labels is 77.50%, 89.68%, and 93.27%, respectively, averaged over both presentation orders with abstentions counted as disagreements.

We report this complementary judge validation in Appendix F.3 and Table 10 (p. 22), together with pairwise agreement between judges. Appendix E.2 gives the scoring protocol (pp. 20–21).

AuthorsOpenReview ↗

A Gentel Reminder

Dear Reviewer TbC8,

Thank you again for your review. Since the author-reviewer discussion is ending soon, we would like to kindly follow up on our clarification.

In brief, the paper studies how to estimate the win probability between two language models under a fixed judge with fewer generated responses and judge calls. It does not involve reward modeling or RLHF. The revised version restructures the introduction and Sections 2-3 around the problem, the method, and the evidence, extends the comparisons across models, fine-tuning checkpoints, datasets, and judges, and adds the requested RewardBench validation.

We would greatly appreciate it if you could take a look at the revised paper and our responses, and let us know whether they address your concerns. We are happy to answer any further questions.

Best regards,
The authors

Reviewer_34D6OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes crossed comparison using coupled sampling to reduce generation and judgment costs while preserving the independent-output win probability. The authors also introduce adaptive stopping and provide theoretical guarantees and matching lower bounds. In their experiments, they demonstrate cost reductions across multiple generator configurations.

Critical strengths
  • This paper proposes an interesting approach to reducing the computational cost of reliable LLM evaluation (LLM-as-a-judge).
  • The proposed method is supported by substantial theoretical analysis, including unbiased estimation, adaptive stopping, and matching lower bounds.
  • The authors demonstrate cost reductions compared with multiple baselines.
Critical weaknesses
  • W1: Insufficient justification of experimental settings and task selection.
    • This paper evaluates the proposed method using relatively small models from the Qwen2.5 family and two datasets, E2E and CommonGen. However, the reasons for selecting these models and tasks are not sufficiently explained.
    • In the context of LLM-as-a-judge, various benchmarks and evaluation tasks are available, making it unclear why evaluating the proposed method only on a limited set of tasks, primarily focused on keyword-to-text generation, is sufficient. I would like the authors to provide a more detailed justification for these choices. If these choices cannot be adequately justified, I recommend conducting additional experiments.
  • W2: Insufficient consideration of evaluation quality and human preferences.
    • This paper focuses on reducing the cost of pairwise evaluation, but it does not examine how well the resulting judgments align with human preferences.
    • In particular, the authors use a single Qwen2.5-7B judge, and it remains unclear how its model-specific preferences and biases affect the evaluation results. The theoretical guarantees concern estimation accuracy with respect to a fixed comparator rather than the quality of the evaluation itself.
    • Since evaluation quality is as important as evaluation efficiency in NLG evaluation, I recommend validating the results through agreement with human judgments or experiments using different judges. Alternatively, these limitations should be explicitly discussed.
  • W3: Limited discussion of applicability to other evaluation frameworks.
    • This paper focuses on estimating pairwise win probabilities. However, various evaluation frameworks are used in practical LLM evaluation, including Elo ratings, absolute scoring, and multiple-response comparisons.
    • The paper lacks sufficient discussion of the extent to which the proposed method can be extended to these evaluation settings or what technical challenges such extensions might involve. I do not necessarily request additional experiments, but I would appreciate a discussion of these possibilities and their implications for future research.
  • W4: Unclear practical implications for real-world LLM evaluation.
    • Although this paper demonstrates reductions in generation and judgment costs, it remains unclear under what conditions the proposed method should be adopted in practical LLM evaluation.
    • In particular, the benefits may depend on factors such as agreement between coupled outputs, the required evaluation precision, and the relative costs of generation and judgment. Based on these findings, I believe that providing concrete recommendations for real-world applications would help clarify the practical contributions of this research.
Questions for authors

Please address the four weaknesses raised above (W1-W4). However, I do not consider these to be fundamental issues with the paper, but rather suggestions for further strengthening its contributions and overall impact.

1 comment

Dear Reviewer 34D6,

Thank you for your careful reading and constructive suggestions. We have expanded the experiments and clarified the practical guidance. We first summarize the main revisions.

  • Sections 4.1–4.3 and Tables 1–2 expand model scale and add fine-tuning checkpoint comparisons.
  • Section 4.5 adds WildBench under four judges, and Appendix F.3 adds RewardBench validation.
  • Section 4.6 and Appendix F.4 connect resource counts to online execution, target precision, and prices.

Next, we respond to each comment.

W1: Insufficient justification of experimental settings and task selection.

E2E covers generation from meaning representations, and CommonGen covers generation from concept sets. They let us compare estimators with fixed prompts, response distributions, and judges. Scoring the recorded response pairs gives an exact target for that pool against which to measure estimation error. We expanded this comparison to Qwen2.5-Instruct 0.5B, 1.5B, and 7B, and Mistral Small 3 24B, under Qwen2.5-7B-Instruct and Claude Haiku 5.5. Crossed 2 (our method, default setting) uses 42.8–59.2% fewer generated responses and 58.8–88.9% fewer distinct judgments than the best independent baseline for each resource, at error tolerance 0.02 and 95% confidence. The setup and results appear in Sections 4.1–4.2 and Table 1 (pp. 5–7).

We also compare epoch 1 and epoch 2 of one Qwen2.5-1.5B-Instruct fine-tuning run at the same temperature, finding savings under both judges on both datasets. Section 4.3 and Table 2 report this comparison (pp. 7–8). For open-ended generation, Mistral 24B on WildBench reduces both resource counts under all four judges, as shown in Section 4.5 and Table 5 (p. 9).

W2: Insufficient consideration of evaluation quality and human preferences.

We address judge dependence through Qwen and Haiku on E2E and CommonGen, and Qwen, Haiku, Sonnet, and GPT on WildBench. Each judge defines its own target win probability, and resource savings appear under every judge in Tables 1, 2, and 5.

We separately validate judge quality against the published preferred-response labels on all 2,985 pairs in the filtered RewardBench v1 split. Agreement is 77.50% for Qwen2.5-7B-Instruct, 89.68% for Claude Haiku 5.5, and 93.27% for Claude Sonnet 5.5, averaging both presentation orders and counting abstentions as disagreements. Table 10 also reports agreement between judges. The protocol is in Appendix E.2 (pp. 20–21), with results in Appendix F.3 and Table 10 (p. 22).

W3: Limited discussion of applicability to other evaluation frameworks.

Pairwise win probabilities provide evidence for rating models such as Elo and Bradley–Terry. A rating-based extension would combine these estimates and propagate their uncertainty through the rating model. For bounded absolute scores, coupled samples can estimate a difference in expected scores, and the general bounded-score stopping rule applies to this difference. Multiple-response evaluation can reuse crossed comparisons through a pairwise decomposition. Direct listwise evaluation requires specifying the joint target and constructing samples that preserve it. The existing pairwise formulation is in Section 2.1 and Eq. (2), and Section 3.1 gives the general bounded-score stopping rule (pp. 2 and 4–5).

W4: Unclear practical implications for real-world LLM evaluation.

For repeated generator comparisons, Crossed 2 offers a procedure that adapts sampling to the observed scores. Agreement skips judgments, while lower score variance reduces sampling. The shared-Gumbel implementation applies to models sharing a token vocabulary, as explained in Section 2.1 and Algorithm 1 in Appendix A (pp. 2 and 14).

An evaluator can choose the target precision and weight generated responses and distinct judgments by their respective prices. Appendix F.4 and Eq. (35) give this cost model (pp. 23–24). For example, on E2E 0.5B at error tolerance 0.01, Crossed 2 has lower cost than Coupled 4 when the judgment-to-response price ratio exceeds about 0.582, and Coupled 4 has lower cost below this threshold.

We also execute generation, judgment, and stopping together online. In the reported run, Crossed 2 certifies error tolerance 0.05 at 95% confidence using 1,280 responses and 304 distinct judgments, versus 1,536 and 1,509 for Independent 2. Elapsed time decreases from 803.1 to 536.7 seconds. Section 4.6 and Table 6 report this evaluation (pp. 9–10).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

We thank the three reviewers for their careful reading and constructive comments. We have revised the presentation and expanded the experiments. The main changes are as follows.

  • The introduction and Figure 1 now explain the independent-output comparison target, why within-pair comparison can be biased, and how crossed comparison preserves the target.
  • Sections 2 and 3 clarify the coupling, score, and stopping-rule notation. A worked example explains the construction, and Appendix A gives the complete algorithm.
  • Sections 4.1–4.3 distinguish baselines from variants and expand the evaluation to four models up to 24B, two judges, and checkpoints from the same fine-tuning run. Tables 1–3 report the resource comparisons.
  • Section 4.4 and Table 4 separate cancellation from score concentration. Section 4.5 and Table 5 add Mistral 24B on WildBench under four judges.
  • Appendix F.3 and Table 10 add RewardBench validation against preferred labels and agreement between judges.
  • Section 4.6 and Table 6 report online evaluation. Appendix F.4 explains how target precision and resource prices affect practical cost.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.