acceptodds
Under review as a conference paper at ICLR 2027

Home-Away Evaluation: A Home-Field Framework for Pairwise LLM Evaluation

Abstract

Static benchmark accuracy summarizes how often a model succeeds, but obscures where its strengths differ from those of another model. We introduce Home-Away Eval, a pairwise evaluation framework that tests whether comparative strengths transfer to new questions. Each model generates questions from Home seeds that it answers correctly and its opponent answers incorrectly. Independent referees validate the questions and establish their answers; both players then answer the accepted questions on both fields. Only an exclusive correct answer earns a point. In our controlled benchmark-exposure experiments, fine-tuning improves static accuracy, but the fine-tuned checkpoints lose to their clean bases on the generated fields we evaluate. Training on the questions a comparison decides, by contrast, wins the match while leaving static accuracy slightly improved. An authorship study shows that each model answers its own questions more accurately than other authors' questions, which motivates reciprocal fields. The fields expose domain-dependent strengths, distinguish question posing from answering, and focus comparative evidence on questions the two models answer differently. Home-Away Eval complements static benchmarks with an explicit account of who challenges whom, on which questions, and with what evidence.

Primary area:
datasets and benchmarks
Keywords:
LLM evaluation, pairwise evaluation, benchmark contamination, question generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Z73KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes "Home-Away." Starting from the performance differences between the two models on benchmark datasets, it identifies the questions each model excels at, has each model generate its own set of questions, and then has both models independently answer both sets of questions to examine whether their relative advantages on existing benchmarks persist on new questions. The specific steps are as follows: First, identify the questions that each model consistently answers correctly while the other model answers incorrectly, designating these as that model’s "Home seeds," and then have that model generate new questions based on them. Both models first attempt to answer the questions to screen candidate problems; then, two external referee models verify whether the questions are valid and novel relative to their source seeds and determine the reference answers. Both models subsequently re-answer the same set of retained questions, with points awarded only for exclusive correct answers, when exactly one model answers correctly. Finally, the total scores of both models are compared, and their scores on questions generated by themselves and by their opponent are tallied separately.

The paper conducted experiments on multiple-choice question answering (MCQA) and mathematics tasks. Experimental results: a. The six model checkpoints fine-tuned directly using benchmark test questions showed improved average accuracy on the benchmarks they were trained on, but all lost to their respective base models when compared on the newly generated questions. b. In the authorship experiment, all three models achieved higher accuracy on their own questions than on questions from other authors; changing the source and combination of questions also affected the rankings. c. The paper also presents a training case: after being trained on decisive questions derived from model comparisons, the fine-tuned model defeated its base model in a subsequent Home-Away match on newly generated questions and achieved slight improvements on some static benchmarks.

Critical strengths
  1. I believe one valuable aspect of this paper is that it proposes actually verifying whether performance on existing benchmarks can be carried over to newly generated questions, and it conducts foundational analysis as well as benchmark fine-tuning experiments. In Section 5.1, all six fine-tuned checkpoints showed improved average accuracy on the benchmarks used for fine-tuning, yet all lost to their respective base models in Home-Away matches on newly generated questions. These results demonstrate that improvements in accuracy on the benchmarks used for fine-tuning alone are insufficient to judge relative performance on newly generated questions; evaluating such fine-tuning requires supplementary testing on new questions.

  2. The paper provides positive evidence that its evaluation results were used for training. In Section 6.3, the authors trained the model using 2,512 decisive questions derived from model comparisons. In a Home-Away match on newly generated questions (where source seeds were not reused and the evaluation questions contained no exact duplicates of the training questions), the fine-tuned model defeated its base model by a score of 134 to 78. This experiment demonstrates that the evaluation questions, in one instance, aided subsequent training, serving as a direct illustration of the paper’s practical utility.

Critical weaknesses
  1. My main question is: What additional value do the two core design elements, Home seeds and reciprocal authorship, bring to the evaluation compared to simpler approaches for obtaining new questions from third-party sources (such as having a third-party model generate new questions or finding similar third-party questions within the community)?

    Lines 97–102 in Section 2 clarify the main differences between this paper and MathDuels: new questions are generated based on the current model pair’s disagreement items, and both models are required to answer the questions generated for this comparison, covering both multiple-choice question answering (MCQA) and mathematics tasks. While these differences are clear, the existing experiments lack direct validation of the added value of these design choices.

    The shared-seed control in Section 5.2 shows that all three models achieve higher accuracy on their own questions, indicating a correlation between question authorship and performance. However, the "Own" and "Others" comparisons involve different question sets, and the authors acknowledge that their difficulty levels differ (lines 294–295). This observation alone is insufficient to demonstrate that reciprocal authorship is more suitable for testing relative advantages than having both models independently answer a shared set of new third-party questions. The Home-Away experiment in this section simultaneously alters both the seed source and the models used for screening, making it impossible to assess the effect of Home seeds in isolation. Since the candidate questions themselves have already been screened based on disagreement between the initial answers, it is also necessary to distinguish between the effects of this step and those specifically attributable to Home seeds.

    Section 5.1 also fails to report the performance of these six fine-tuned checkpoints on a fixed test set of the same type that was not used for this fine-tuning (i.e., a set of similar third-party new questions). Therefore, it remains unclear whether they exhibit a more general decline in performance, and whether testing on ordinary new questions would yield the same results.

    I would like to see a direct comparison involving a representative model pair:

    a. Randomly sample an equal number of ordinary seeds from the same source benchmark, maintaining consistency in the question-generating models, generation budget, screening, and referee validation processes, and compare them with Home seeds.

    b. With fixed seeds, generation budgets, and subsequent screening and referee validation processes, replace the question-generating models with an independent third-party model and compare the results of both approaches.

    c. If a community-based third-party benchmark of similar new questions exists, I would also like to see supplementary evaluations conducted.

    This would help determine whether the paper’s key design provides more insight than ordinary testing on new questions, and it directly influences my assessment of the method’s contributions.

  2. For matches with a small number of questions and close scores, the current results are insufficient to demonstrate whether the outcomes remain stable across independent runs.

    In Section 5.2, Gemma’s victory over Qwen was based on only 26 accepted questions, with a lead of two points (lines 296–297). The paper also explicitly states in Section 7 that it does not report variation across independent field-generation runs (lines 474–475). Section 5.3 demonstrates how the score gap widens after increasing the budget during a single generation run, but this does not indicate whether the same outcome would be obtained after independently regenerating the questions under the same rules.

    For such small margins, independent field generation and answering must be repeated under identical settings, and changes in scores and outcomes must be reported. Otherwise, it is currently impossible to determine whether the observed lead persists across independent runs or whether it is susceptible to variation due to the randomness of question selection and answering. The evaluation paper needs to address this for reliability.

Questions for authors
  1. What conclusions can actually be drawn from winning a Home-Away match? Please clarify the criteria used in the paper to determine whether "the relative advantage on the source questions persists on the new questions," and specify exactly what that advantage entails.

    Section 3.5, lines 169–172, states that the final comparison implicitly incorporates question-generation ability. Therefore, it is necessary to clarify the correspondence between winning on the question sets generated and screened by both models and the persistence of the answering advantage observed on the source questions. Please explain which experimental results directly test whether the advantage persists, and what scope of capability assessments the final match outcome supports. Section 6.2 further indicates that generated questions do not necessarily retain the subject category of the source questions.

  2. In Section 5.2, all three models achieve higher accuracy on their own question sets; however, the "Own" and "Others" comparisons involve different question sets, and the authors acknowledge that the difficulty levels of these sets differ (lines 294–295). Could you provide the complete accuracy rates of the three answering models on the three author-created question sets, and explain whether the current analysis can distinguish between the overall difficulty of the question sets, differences in content, and the models’ relative advantage on their own questions? This would help clarify which conclusions the authorship experiment can support.

  3. Inconsistent statements. How exactly were the eight few-shot examples for the generation phase selected? Lines 144–145 of Section 3.3 state that the examples came from their respective Home pools; however, lines 597 and 602–606 of Appendix A state that the examples came from data splits disjoint from the evaluated benchmark items and were fixed per benchmark. Please clarify to which stages the description "fixed per benchmark" applies; do all reference examples during generation come from the generating model’s Home pool for the current model pair, or are fixed reference examples used, with only the current source seed coming from the Home pool? Is the current source seed counted among the eight examples? Please explain how the descriptions in the main text and the appendix correspond, as these settings correspond to different generation conditions.

  4. Why do the two rankings in Table 2 use different task scopes? Lines 229–230 state that the static ranking averages five benchmarks, while the Home-Away ranking aggregates only multiple-choice question answering (MCQA) matches. According to the values in the table, when averaging only three MCQA benchmarks, the static ranking for Qwen3.5-9B (contam-all) is 2nd, not 1st. Can you compare and explain the ranking changes under the same task scope?

  5. What are the core value and ultimate contribution of this evaluation method from an application perspective? For example, is it to help users choose between two candidate models? To analyze the respective strengths and weaknesses of the models? Or to select training data and provide guidance for subsequent fine-tuning? Or to provide updated, more challenging evaluation data? Its current purpose is not yet clear.

Reviewer_Cy9BOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes Home-Away Evaluation, a dynamic framework for pairwise LLM evaluation. The method first identifies questions that one model answers correctly while the other answers incorrectly. Each model then generates new questions based on its own strengths, and both models answer the validated questions to determine the winner. Through experiments on benchmark contamination, question authorship, and model rankings, the authors aim to show that this approach can reveal capability differences that traditional static benchmarks may overlook.

Critical strengths
  1. The research problem is meaningful. Overall accuracy on static benchmarks may hide differences in the specific questions that models can solve. This paper attempts to examine whether a model's strengths on existing benchmarks transfer to newly generated questions, which is a worthwhile research direction.

  2. The method is intuitive and clearly presented. The home-away mechanism allows both models to generate questions based on their respective strengths, rather than relying entirely on a fixed benchmark or a single question generator. The authors also introduce a dual-referee validation process to reduce errors and ambiguity in generated questions.

  3. The experiments cover a relatively broad range of settings. The authors evaluate the framework on MMLU, MMLU-Pro, GPQA, GSM8K, and MATH, covering different models, model sizes, and contamination settings. They also investigate authorship bias, referee agreement, question generation budgets, and training on decisive questions.

Critical weaknesses
  1. Limited novelty and substantial overlap with prior work. Automatic question generation, reciprocal model evaluation, opponent-targeted question generation, and disagreement-based evaluation have already been explored in previous studies. For example, SKATE, MathDuels, PeerRank, and LivingArena have investigated adversarial question generation, home-field advantages, or reciprocal evaluation. The main novelty of this paper appears to be using pairwise exclusive-correct questions from existing benchmarks as generation seeds and combining them with home-away scoring. This seems more like a combination of existing ideas than a substantially new evaluation framework. The paper also lacks direct experimental comparisons with these closely related methods, making it difficult to assess its unique contribution.

  2. The question selection process may artificially amplify model differences, and the final score is difficult to interpret. The authors first select seeds based on model disagreements, then only validate generated questions when the models give different answers. Therefore, the evaluation set is already biased toward highlighting their differences. For example, the proportion of decisive questions on MMLU increases from 10.6% to 65.1%, which may largely result from this selection process rather than better capability measurement. Moreover, accepted question counts are not equalized and scores are not normalized, mixing question-generation ability with answering performance.

  3. The contamination experiment results need further explanation. Fine-tuning improves static benchmark accuracy, yet the fine-tuned Qwen3.5-9B loses 2–21 to its original version in Home-Away evaluation. While fine-tuning can cause some capability degradation, such a large reversal is surprising. Since the generated questions are specifically conditioned on differences between the two checkpoints, the method may amplify localized degradation. These results do not establish that the fine-tuned model performs worse on general unseen questions or that Home-Away is more reliable than an independent test set.

  4. The reliability of generated questions and referees is questionable. The authors use GPT-5.6 Terra and Gemini-3.6-Flash with low reasoning effort as referees, without sufficiently demonstrating their ability to validate challenging questions. Referee agreement does not guarantee correct or unambiguous answers. Among questions accepted by both referees, answer disagreement reaches 64.5% on GSM8K, 38.3% on MATH, and 52.0% on Global-Math. Human-referee agreement on mathematical answers is also only 74.8%. Stronger referees or more extensive human validation would help establish the reliability of the evaluation results.

  5. Model rankings are sensitive to question generation and lack stability analysis. Gemma-4 ranks third in static mathematical accuracy, fourth under aggregated Home-Away evaluation, but first under Global-Math. Such large differences suggest that rankings may depend heavily on how questions are generated rather than reflecting stable model capabilities. Similarly, the reported home-field advantage may be influenced by differences in question difficulty and distribution. The paper does not report ranking variance across independently generated evaluation sets, making the reliability of these rankings difficult to assess.

Questions for authors

Please see weaknesses.

Reviewer_zg4cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Home-Away Evaluation, a framework for comparing two LLMs on newly generated questions. The authors first select benchmark items that one model consistently answers correctly and its opponent answers incorrectly. These items form each model's Home seed pool. Both models then generate new questions from their respective pools. Independent referees validate the questions and establish reference answers. Both models answer all retained questions afresh. A model earns a point only when it alone answers correctly. Experiments cover multiple-choice and mathematical tasks. They examine how benchmark exposure, question authorship, generation budgets, and referee filtering affect match outcomes.

Critical strengths
  1. The paper addresses a practically relevant problem. Similar scores on static benchmarks do not imply identical capability profiles. Comparing models on questions generated by both participants can reveal relative strengths that a single accuracy score does not capture.

  2. The evaluation procedure is clearly described. The authors discard the question generator's proposed answers and use independent referees to establish reference answers. Both models then answer the questions afresh. The experiments also examine the effects of question authorship and referee filtering, helping readers understand the factors behind match scores.

Critical weaknesses
  1. Insufficient comparison with directly related work.

    Eval Duel and the same team's SKATE already study mutual question generation and cross-answering, using verifiable answers to compare model capabilities. SKATE also analyzes models' advantages on their own questions and differences between question-generation and answering ability. Home-Away neither cites nor discusses these works. Its related-work comparison therefore omits studies that directly overlap with its core framework.

    The clearest differences are its use of benchmark items with directional correctness disagreements as seeds and independent referee validation of generated natural-language questions. Compared with the cited MathDuels, Home-Away constructs question sets for specific model pairs and has both participants answer their own generated questions. These differences warrant investigation. However, the paper needs to define its contributions around these specific designs and support them with comparative evidence.

  2. The value of the Home seed selection rule is not directly established.

    The rule selects questions that a model consistently answers correctly while its opponent answers incorrectly. This is an important distinction from general self-generated evaluation, but the experiments do not isolate its contribution.

    Please add comparisons using random seeds, seeds selected only for the model's own correctness, and disagreement seeds without a directional requirement. Relevant metrics include the number of valid, discriminative questions per unit of generation budget and the stability of match outcomes. Without these comparisons, it is difficult to determine how much of the observed discrimination comes from Home seeds. The results do not separate this contribution from generating new questions and filtering model disagreements, both of which are existing mechanisms.

Questions for authors
  1. How can gains from benchmark memorization be distinguished from other effects of fine-tuning?

    How do these fine-tuned models perform on ordinary unseen questions? With the same fine-tuning procedure and an equal amount of data without benchmark exposure, would they also lose to their original base models? How much of the reversal in match outcomes comes from benchmark memorization? How much comes from the effects of fine-tuning on generalization or question-generation ability?

  2. What is the main novelty relative to Eval Duel and SKATE?

    Mutual question generation and cross-answering already have direct precedents. Are Home seeds the main new design element introduced in this paper? If so, which experiments show their advantages over existing question-generation strategies?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion and revisions

We thank the reviewers for a constructive discussion. The main points and our answers were:

  • Novelty (zg4c, Cy9B): mutual generation and cross-answering follow prior duel-style work; Home-Away adds directional Home seeds, acceptance only when two referees agree on a question's validity and answer, and two-sided exclusive-correct points.
  • Value of Home seeds (zg4c, Cy9B, Z73K): at equal budgets, Home seeds nearly double the decisive yield of random seeds and raise the author's share of decisive points from 54% to 67%; a SKATE-style targeting prompt matches the yield but not the share, and a third-party writer preserves 5 of 6 winners.
  • Contamination (zg4c, Cy9B, Z73K): benchmark gains need not transfer: contaminated checkpoints stay within 1.1 points of their bases on MMLU-CF and lose their matches. Gemma-4-E4B-it fine-tunes on fixed schedules and on clean data also lose, so a loss shows worse performance on new questions, not exposure, while decisive-question training wins. Appendix A misstated the training schedule: checkpoints were selected by test-set accuracy.
  • Referees (Cy9B): stronger referees retain fewer questions but agree with our answer keys on 99.6% (MCQA) and 99.2% (mathematics) of them and reverse no winner.
  • Close matches and rankings (Cy9B, Z73K): Gemma-4's 10–8 win over Qwen3.6 is within noise. We proposed a criterion for declaring a winner (at least 10 decisive questions and 90% retention over 2,000 seed resamples), under which no two runs of the same MMLU match declare opposite winners, and we report rank intervals.
  • Authorship (Z73K): an additive fit separates answerer and field difficulty; own-field advantages remain for Gemma-4 and GPT-OSS but not for Qwen3.6.

Revisions. The revised paper incorporates these results; changes are in blue, and what does not fit in 10 pages is in the Appendix. Section 2 adds a table, so submitted Tables 1–6 are now Tables 2–7 and Tables 7–13 are Tables 10, 11, 13, and 18–21; main-text sections and figures keep their numbers, and the former Appendix E is now Appendix F.

  • Positioning (zg4c W1/Q2 and follow-up Q1, Cy9B W1): Section 2 and Table 1 compare SKATE (Eval Duel), MathDuels, PeerRank, and LivingArena; Section 1 restates the contributions.
  • Seed rules and writers (zg4c W1/Q2 and W2, Cy9B W1, Z73K W1): Section 5.3, Tables 14–15, and Appendix E (with three MMLU-Pro pairs).
  • Contamination (zg4c Q1 and follow-up Q2–Q3, Cy9B W3, Z73K W1 and follow-up Q1–Q2): Tables 2 and 7 are rescored on all generated questions with fresh answers under dual-referee agreement, and Gemma-4-E4B-it's mathematics match is regenerated at 120 questions per seed; totals change, and no winner does. Section 5.1, Tables 8–9, and Appendix A add the held-out and perturbed sets, the fixed-schedule and clean-data controls, and the corrected training description; Table 7 adds a fixed 22-epoch decisive-question control.
  • Referees (Cy9B W4): Section 6.1 and Table 12.
  • Uncertainty (Cy9B W5, Z73K W2 and follow-up Q3–Q4): Section 3.5 adds the criterion; tables report match status (e.g., Tables 3, 9, and 15), and Table 22 lists the MMLU matches uncertain in the original run or a rerun, with larger runs. Sections 5.2, 5.3, and 7 cover the 10–8, the closest pair across runs, and the regeneration of all 15 MMLU matches; Table 17 gives rank intervals, and Figure 2 marks Gemma-4's reversal over Qwen3.5-35B as uncertain.
  • Authorship and scope (Cy9B W2 and W5, Z73K Q2–Q4): the additive fit (Section 5.2) and all 9 answerer-by-author accuracies (Table 16); MCQA-scope static ranks and declared-only records in Table 3; the few-shot examples (Section 3.3, Appendix A); a screening audit and equal field weighting (Sections 3.4–3.5).
  • Table 3's records, Section 5.5, and Tables 20–21 use the same rescoring; Table 3's ranks are unchanged. Table 13 replaces one example.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.