- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes "Home-Away." Starting from the performance differences between the two models on benchmark datasets, it identifies the questions each model excels at, has each model generate its own set of questions, and then has both models independently answer both sets of questions to examine whether their relative advantages on existing benchmarks persist on new questions. The specific steps are as follows: First, identify the questions that each model consistently answers correctly while the other model answers incorrectly, designating these as that model’s "Home seeds," and then have that model generate new questions based on them. Both models first attempt to answer the questions to screen candidate problems; then, two external referee models verify whether the questions are valid and novel relative to their source seeds and determine the reference answers. Both models subsequently re-answer the same set of retained questions, with points awarded only for exclusive correct answers, when exactly one model answers correctly. Finally, the total scores of both models are compared, and their scores on questions generated by themselves and by their opponent are tallied separately.
The paper conducted experiments on multiple-choice question answering (MCQA) and mathematics tasks. Experimental results: a. The six model checkpoints fine-tuned directly using benchmark test questions showed improved average accuracy on the benchmarks they were trained on, but all lost to their respective base models when compared on the newly generated questions. b. In the authorship experiment, all three models achieved higher accuracy on their own questions than on questions from other authors; changing the source and combination of questions also affected the rankings. c. The paper also presents a training case: after being trained on decisive questions derived from model comparisons, the fine-tuned model defeated its base model in a subsequent Home-Away match on newly generated questions and achieved slight improvements on some static benchmarks.
Critical strengths
-
I believe one valuable aspect of this paper is that it proposes actually verifying whether performance on existing benchmarks can be carried over to newly generated questions, and it conducts foundational analysis as well as benchmark fine-tuning experiments. In Section 5.1, all six fine-tuned checkpoints showed improved average accuracy on the benchmarks used for fine-tuning, yet all lost to their respective base models in Home-Away matches on newly generated questions. These results demonstrate that improvements in accuracy on the benchmarks used for fine-tuning alone are insufficient to judge relative performance on newly generated questions; evaluating such fine-tuning requires supplementary testing on new questions.
-
The paper provides positive evidence that its evaluation results were used for training. In Section 6.3, the authors trained the model using 2,512 decisive questions derived from model comparisons. In a Home-Away match on newly generated questions (where source seeds were not reused and the evaluation questions contained no exact duplicates of the training questions), the fine-tuned model defeated its base model by a score of 134 to 78. This experiment demonstrates that the evaluation questions, in one instance, aided subsequent training, serving as a direct illustration of the paper’s practical utility.
Critical weaknesses
-
My main question is: What additional value do the two core design elements, Home seeds and reciprocal authorship, bring to the evaluation compared to simpler approaches for obtaining new questions from third-party sources (such as having a third-party model generate new questions or finding similar third-party questions within the community)?
Lines 97–102 in Section 2 clarify the main differences between this paper and MathDuels: new questions are generated based on the current model pair’s disagreement items, and both models are required to answer the questions generated for this comparison, covering both multiple-choice question answering (MCQA) and mathematics tasks. While these differences are clear, the existing experiments lack direct validation of the added value of these design choices.
The shared-seed control in Section 5.2 shows that all three models achieve higher accuracy on their own questions, indicating a correlation between question authorship and performance. However, the "Own" and "Others" comparisons involve different question sets, and the authors acknowledge that their difficulty levels differ (lines 294–295). This observation alone is insufficient to demonstrate that reciprocal authorship is more suitable for testing relative advantages than having both models independently answer a shared set of new third-party questions. The Home-Away experiment in this section simultaneously alters both the seed source and the models used for screening, making it impossible to assess the effect of Home seeds in isolation. Since the candidate questions themselves have already been screened based on disagreement between the initial answers, it is also necessary to distinguish between the effects of this step and those specifically attributable to Home seeds.
Section 5.1 also fails to report the performance of these six fine-tuned checkpoints on a fixed test set of the same type that was not used for this fine-tuning (i.e., a set of similar third-party new questions). Therefore, it remains unclear whether they exhibit a more general decline in performance, and whether testing on ordinary new questions would yield the same results.
I would like to see a direct comparison involving a representative model pair:
a. Randomly sample an equal number of ordinary seeds from the same source benchmark, maintaining consistency in the question-generating models, generation budget, screening, and referee validation processes, and compare them with Home seeds.
b. With fixed seeds, generation budgets, and subsequent screening and referee validation processes, replace the question-generating models with an independent third-party model and compare the results of both approaches.
c. If a community-based third-party benchmark of similar new questions exists, I would also like to see supplementary evaluations conducted.
This would help determine whether the paper’s key design provides more insight than ordinary testing on new questions, and it directly influences my assessment of the method’s contributions.
-
For matches with a small number of questions and close scores, the current results are insufficient to demonstrate whether the outcomes remain stable across independent runs.
In Section 5.2, Gemma’s victory over Qwen was based on only 26 accepted questions, with a lead of two points (lines 296–297). The paper also explicitly states in Section 7 that it does not report variation across independent field-generation runs (lines 474–475). Section 5.3 demonstrates how the score gap widens after increasing the budget during a single generation run, but this does not indicate whether the same outcome would be obtained after independently regenerating the questions under the same rules.
For such small margins, independent field generation and answering must be repeated under identical settings, and changes in scores and outcomes must be reported. Otherwise, it is currently impossible to determine whether the observed lead persists across independent runs or whether it is susceptible to variation due to the randomness of question selection and answering. The evaluation paper needs to address this for reliability.
Questions for authors
-
What conclusions can actually be drawn from winning a Home-Away match? Please clarify the criteria used in the paper to determine whether "the relative advantage on the source questions persists on the new questions," and specify exactly what that advantage entails.
Section 3.5, lines 169–172, states that the final comparison implicitly incorporates question-generation ability. Therefore, it is necessary to clarify the correspondence between winning on the question sets generated and screened by both models and the persistence of the answering advantage observed on the source questions. Please explain which experimental results directly test whether the advantage persists, and what scope of capability assessments the final match outcome supports. Section 6.2 further indicates that generated questions do not necessarily retain the subject category of the source questions.
-
In Section 5.2, all three models achieve higher accuracy on their own question sets; however, the "Own" and "Others" comparisons involve different question sets, and the authors acknowledge that the difficulty levels of these sets differ (lines 294–295). Could you provide the complete accuracy rates of the three answering models on the three author-created question sets, and explain whether the current analysis can distinguish between the overall difficulty of the question sets, differences in content, and the models’ relative advantage on their own questions? This would help clarify which conclusions the authorship experiment can support.
-
Inconsistent statements. How exactly were the eight few-shot examples for the generation phase selected? Lines 144–145 of Section 3.3 state that the examples came from their respective Home pools; however, lines 597 and 602–606 of Appendix A state that the examples came from data splits disjoint from the evaluated benchmark items and were fixed per benchmark. Please clarify to which stages the description "fixed per benchmark" applies; do all reference examples during generation come from the generating model’s Home pool for the current model pair, or are fixed reference examples used, with only the current source seed coming from the Home pool? Is the current source seed counted among the eight examples? Please explain how the descriptions in the main text and the appendix correspond, as these settings correspond to different generation conditions.
-
Why do the two rankings in Table 2 use different task scopes? Lines 229–230 state that the static ranking averages five benchmarks, while the Home-Away ranking aggregates only multiple-choice question answering (MCQA) matches. According to the values in the table, when averaging only three MCQA benchmarks, the static ranking for Qwen3.5-9B (contam-all) is 2nd, not 1st. Can you compare and explain the ranking changes under the same task scope?
-
What are the core value and ultimate contribution of this evaluation method from an application perspective? For example, is it to help users choose between two candidate models? To analyze the respective strengths and weaknesses of the models? Or to select training data and provide guidance for subsequent fine-tuning? Or to provide updated, more challenging evaluation data? Its current purpose is not yet clear.
Summary of the discussion and revisions
We thank the reviewers for a constructive discussion. The main points and our answers were:
Revisions. The revised paper incorporates these results; changes are in blue, and what does not fit in 10 pages is in the Appendix. Section 2 adds a table, so submitted Tables 1–6 are now Tables 2–7 and Tables 7–13 are Tables 10, 11, 13, and 18–21; main-text sections and figures keep their numbers, and the former Appendix E is now Appendix F.