acceptodds
Under review as a conference paper at ICLR 2027

How Much of a Language Model Group's Collective Failure Is Visible in Its Pairs?

Abstract

How much do the pairwise failure rates of the models in a language-model group reveal about them failing together? Individual and pairwise rates, which form a pairwise table, do not in general identify the collective failure rate, the share of items on which every model fails. However, across 48 public evaluation pools from five releases, completions of the pairwise table approximate the collective failure rate closely. For groups of eight models, the range of collective failure rates allowed by the pairwise table is on average 16 percentage points wide, more than twice a group's observed rate, so the table alone pins that rate down only to within ±120% of its value. Yet the least-structured completion, pairwise maximum entropy, reconstructs that rate with 15.9% mean relative error (1.1 percentage points), and a Gaussian copula gives the same qualitative result. Where pools have enough items to measure it, most of the dependence among model failures is pairwise. The regularity has limits: error rises with group size and on held-out items when collective failures are scarce. One use follows. Across the 48 pools, groups chosen by maximum entropy from a pairwise table fail together less often on held-out items than groups of the most accurate models, on average and in most pools (3.3 percentage points at eight models), also when the choice is restricted to the strongest candidates. Exploratory studies on SWE-bench Verified and RewardBench 2 are consistent with this. The results hold within an evaluation, not across evaluations.

Primary area:
datasets and benchmarks
Keywords:
LLM evaluation, ensemble selection, co-failure, maximum entropy, partial identification, model routing, LLM-as-a-judge, benchmarks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pDgxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether pairwise co-failure statistics of language models can estimate the collective all-failure rate, the probability that every model in a group fails on one input. Theoretically, pairwise statistics only constrain this rate within an interval and cannot uniquely determine it. The authors find that the maximum-entropy method produces strong approximations, and that for the group sizes studied, most of the measured failure dependence is captured by pairwise terms.

Critical strengths

The empirical scope is broad, spanning substantial model pools and benchmarks, and the findings are connected to a principled maximum-entropy framework.

Critical weaknesses

(1) The motivation is unclear. The estimation in the main experiments seems to require complete test results, which are then compressed into a pairwise table and used to reconstruct the joint failure probability. If the complete results are already available, it is unclear why this extra step is needed. Although the paper also studies incomplete logs, the estimation errors appear to remain substantial, leaving the practical benefits insufficiently established. In addition, the paper’s own experiments also show that the estimate does not transfer well across benchmarks. The method performs substantially worse in held-out estimation than in reconstruction, and since it relies on pool-specific pairwise error statistics, its ability to transfer across benchmarks or datasets appears limited and may require additional target-distribution measurements—limiting its practical applicability when such measurements are costly or unavailable.

(2) The “mostly pairwise dependence” explanation is only partially validated: the interaction-order analysis covers 13 pools at k=6 and only 3 MMLU-family pools at k=8, leaving higher-order effects insufficiently characterized on open-ended, code, and agentic tasks.

Questions for authors

Please see weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer pDgx (1/2)

Thank you for this comment. We reply to weakness (1) here and to weakness (2) in a second comment.

We agree that when the complete response matrix is available, collective failure can be counted directly. We are not proposing maximum entropy as a replacement for that count. Our question is different: how much information about collective failure is lost when model evaluations are summarized by individual and pairwise failure rates?

This matters because correlated errors across language models are already studied and reported using pairwise statistics (Kim et al., 2025; Goel et al., 2025). We use complete response matrices as ground truth to test how informative such summaries are. At eight models, the pairwise table allows a range of collective failure rates averaging 15.8 percentage points in width. Yet maximum entropy reconstructs the observed rate with only 1.1 percentage points of mean absolute error (15.9% relative error; Section 3). The point is not that maximum entropy beats direct counting, but that much of the information about collective failure remains in the pairs.

We also test whether that information is useful for choosing model groups. At eight models, selecting groups using pairwise statistics reduces held-out collective failure by 3.3 percentage points compared with choosing the most accurate models. The reduction is 2.0 points when selection is restricted to strong candidates (44 pools), and 2.2 points when the selected group's mean accuracy can fall by at most two points on the fit items (Section 5; Appendix B lists the pools).

For group selection, the full response matrix gives only a small additional benefit. In a separate test where every method chooses from the same set of up to 300 candidate groups, the group chosen from the pairwise table fails together on 10.03% of held-out items at eight models, the group chosen from the full response matrix on 9.95%, and the most accurate candidate group on 11.58%. The difference between pairwise selection and direct counting is not statistically resolved (Appendix G). So the pairs keep almost all of the benefit of the full data. On SWE-bench Verified, groups chosen from pairs also fail together on 9–15% fewer held-out tasks than the most accurate systems, though not at eight systems in its one-agent subset (Section 5, Appendix H).

Prediction on new items is harder, as you note. At eight models, maximum entropy misses by 32.4%. Direct counting on the full fit-half data misses by almost as much (30.3%). So even the full data does not fix this, and both errors drop sharply when the fit half has more collective failures (Section 3, Figure 3). With incomplete records, pairs do better than counting on the few items every model shares, but scoring all models on one common set of items is better when that is possible (Section 3, Appendix L).

We agree that pairwise statistics do not reliably transfer across benchmarks (Section 7, Appendix L), and we do not claim otherwise. The value we demonstrate is narrower: within an evaluation, pairwise statistics retain enough information to select useful model groups without access to the full response matrix.

One possible use is for benchmark owners to publish pairwise statistics without releasing item-level responses (Section 7). A single such table could then be used to study many possible model groups. We have tested what remains possible under this information restriction, but have not demonstrated an actual deployment with pairwise-only access or a reduction in evaluation costs.

So the paper measures what pairwise statistics keep and where they fall short: the selection results on new items show that what they keep is useful, and prediction on new items, incomplete records and transfer show its limits.

What we changed in the revised PDF for this comment

  • Section 5, Table 15 and Appendix G: a new test that limits how much accuracy the chosen group may give up (2.2 points at a two-point limit). The abstract now gives this result and the 2.0-point result for strong candidates.
  • Appendix K and Table 24: a new comparison of maximum entropy and direct counting when only a few fit items are available.
  • Section 7: the transfer paragraph now gives the cross-benchmark result in more detail.
  • Appendix B: a new list of every pool subset with its rule, for example the 44 pools of the strong-candidate setting.
AuthorsOpenReview ↗

Response to Reviewer pDgx (2/2)

Weakness (2). Thanks, this is a fair point. The analysis was narrow because it needs a lot of items. Measuring every order of dependence among k models requires enough items to estimate the probabilities of all 2^k failure patterns reliably, so we kept the item rule from the submitted paper: at least 40 × 2^k items (2,560 at six models, 10,240 at eight). At eight models only three pools, all MMLU-type, are that big.

So we ran the analysis on every pool the rule allows at smaller group sizes: 36, 28 and 22 pools at three, four and five models. Pairs still carry about 96 to 97% of the dependence on average, and no single pool goes below about 90% (Section 4, Table 10). This includes open-ended tasks: open-ended math and other open-ended generation both stay above 94% (Table 22). For agentic coding, SWE-bench Verified and SWE-bench Pro are only big enough for three models, and there pairs carry about 99% (Appendix F).

Code-generation pools are too small for this analysis at any size (50 to 78 items), and the agentic pools are too small beyond three models. There we checked the thing the analysis is meant to explain, collective failure itself. At eight models, maximum entropy has 8.3% reconstruction error on code generation and 8.0% on SWE-bench Verified. On SWE-bench, the range the pairwise table alone allows is far wider: its half-width is nearly six times that error (Appendix H, Table 22). These results extend the reconstruction evidence to coding tasks, although we still cannot measure the full eight-model breakdown by interaction order on them.

We agree the result is descriptive and limited to the group sizes the data supports, and the paper says so (Section 4).

What changed in the revised PDF:

  • Section 4 now states the item rule in the main text and extends the analysis to every pool it allows, including open-ended tasks and agentic coding.
  • New Table 10 (Appendix F) with the shares on all these pools.
  • New Table 22 with results by answer format.

Across both comments: pairs carry most of the measured dependence on every pool the data allows, and the limits you raise, held-out error and no transfer across benchmarks, are stated in the paper as limits.

Thank you for the detailed response. The results demonstrate that pairwise statistics preserve substantial information about collective failure, and the additional group-selection experiments help clarify this finding. However, the motivation and practical value are still unclear to me. First, the authors mention that benchmark owners could release pairwise statistics instead of item-level results, but it is unclear how common this scenario is or how useful the method would be in practice. Even if only these pairwise statistics are available, they may have been collected under inconsistent evaluation conditions—for example, different decoding settings for the same model across pairs. Given that these conditions may affect the failure statistics, how can such summaries be correctly combined into a joint model. Thus, I maintain my original score.

AuthorsOpenReview ↗

Reply to pDgx follow-up: motivation, practical value and evaluation conditions

Thank you for following up. We think we could have explained the motivation of our work more clearly.

Our main contribution is an empirical finding about how language models fail together, rather than a new estimation method. Suppose a future study develops a method based on the assumption that pairwise failure statistics preserve substantial information about larger groups. That assumption would itself need substantial empirical evidence. Providing that evidence is the purpose of our paper. We use maximum entropy as a tool to study this question across 48 evaluation pools and 13,877 model groups. The experiments on model families, item difficulty, interaction orders, and held-out prediction examine how robust this finding is and where it breaks down.

Regarding practical value, there are already studies that release pairwise statistics. For instance, Kim et al. (2025) publish individual model accuracies and pairwise outcome statistics for HELM and Hugging Face models (https://github.com/nikhgarg/llm_correlated_errors_public).

Consider a benchmark such as FrontierMath. Its questions are unpublished, which reduces the chance that models have seen them in training, and model traces can be viewed only for the 10 public questions (https://epoch.ai/benchmarks/about). When only each model's score is available for the private questions, researchers can tell which models perform better, but not whether two models fail on the same questions or complement each other. Pairwise failure statistics would fill this gap. The benchmark owner can compute them from the private results and release them without revealing any question. The community could then study many different combinations of the evaluated models without access to the per-question results.

We are not claiming that this is already common practice. Rather, we encourage benchmark owners to release these statistics alongside individual accuracies, and our paper provides empirical evidence for why such a release could be valuable. One use is choosing groups of models that collectively solve more questions. The finding itself rests on the reconstruction results (Section 3); Section 5 instead tests whether it is useful in practice: the selection algorithm only uses pairwise statistics, and we retain the full response matrix as ground truth to evaluate the selected groups on held-out items. This is a measured result, not a projection: at eight models, the selected groups fail together 3.3 percentage points less often on held-out items than the most accurate models, and in a separate test about as often as groups selected from the full response matrix (10.03% against 9.95%, Appendix G).

We focused the main paper on establishing the empirical finding and examining its robustness and limitations. This is why we discussed the potential use of pairwise releases primarily in Section 7. From this discussion, however, we realize that this motivation deserves more attention. Specifically, we will explain the motivation for releasing pairwise statistics in Section 1, clarify how the coverage experiment in Section 5 simulates this setting, and expand Section 7 to discuss what such releases would enable. Given the limited time remaining in the discussion period, we plan to incorporate these clarifications in the camera-ready version.

Regarding inconsistent evaluation conditions, each evaluation pool in our study consists of multiple models evaluated on the same set of benchmark items. To illustrate the consistency issue with a simple example, suppose we let Model A answer question 1, change its decoding settings before it answers question 2, and continue doing so across the dataset. We would then be mixing results from different configurations of Model A in a single response vector. A similar problem arises in your example of using different configurations of Model A for different pairs.

Our setting is different. We start from the evaluation results of Model A under a given configuration across the dataset. We use this same response vector to compute its individual failure rate, its co-failure rate with Model B, its co-failure rate with Model C, and so on. In other words, the pairwise statistics are computed from a common response matrix after evaluation, rather than obtained by evaluating each pair separately. Different decoding configurations should be treated as separate systems. We agree that combining pairwise statistics from inconsistent evaluation conditions would be problematic, but that is not the setting we study.

We appreciate this discussion, as it helped us identify where the motivation of our paper needs to be communicated more clearly. Our main aim remains to establish an empirical finding about collective language-model behavior, with the coverage experiment demonstrating one use of that finding.

Reviewer_6gpyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether the collective failure rate of a group of language models—the fraction of items on which every model fails—can be inferred from individual failure rates and pairwise co-failure rates. The authors first use a linear program to show that these first- and second-order statistics do not identify collective failure in general. They then evaluate pairwise maximum-entropy and Gaussian-copula completions across 48 public evaluation pools, analyze how much dependence is carried by pairwise interactions, and use pairwise maximum entropy to select model groups with lower held-out collective failure.

Critical strengths

S1. The paper addresses an interesting and potentially useful problem: using pairwise failure summaries to reason about groups of more than two models.

S2. The paper clearly separates theoretical non-identifiability from empirical reconstructability by comparing exact compatible intervals with practical estimators.

S3. The group-selection experiments demonstrate a plausible application of pairwise failure information for building coverage-oriented model ensembles.

Critical weaknesses

W1. The abstract and introduction introduce many concepts before defining them, including pairwise table, completion, compatible interval, coverage, reconstruction, held-out prediction, and the strong-quarter rule, making the paper difficult to read.

W2. The main text gives too little methodological detail about the maximum-entropy and Gaussian-copula estimators, including their objectives, constraints, fitting procedures, numerical approximations, and treatment of invalid correlation matrices.

W3. Although maximum entropy estimates collective failure reasonably well, its benefit for model selection appears marginal under stronger and more realistic candidate restrictions.

W4. The analysis is largely independent of language models: it studies generic binary failure indicators and does not relate failure dependence to model properties such as scale, architecture, family, or training-data overlap.

W5. The paper does not examine how failure dependence varies with item properties such as domain, difficulty, format, or ambiguity, limiting its ability to explain when and why models fail together.

W6. The finding that pairwise terms explain most dependence is descriptive and limited to relatively small groups in data-rich pools; the paper does not provide a mechanism or predictive theory connecting this regularity to language-model behavior.

Questions for authors

Q1. Can the authors stratify the analysis by model family, scale, architecture, pretraining-source overlap, and benchmark domain or difficulty to determine whether the pairwise regularity is driven by identifiable properties of models or items?

Q2. Would conditional or structured estimators that incorporate model and item metadata improve collective-failure estimation or group selection compared with the purely aggregate pairwise table?

Q3. Can the authors provide evidence or a theoretical argument explaining why pairwise interactions dominate in these evaluation pools, and clarify when this pattern should be expected to fail for larger or more heterogeneous model groups?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6gpy (1/2)

Thanks a lot for the detailed review. We reply in two comments: W1 to W4 here, and W5, W6, Q2 and the list of changes in the second.

W1 (terms used before they are defined). Fair point, and we fixed it. The abstract and introduction now define the main terms when they are introduced, including the strong-quarter rule ("only each pool's most accurate quarter of models"). We also rewrote Sections 2 and 3 in plainer language.

W2 (too little detail on the estimators). Agreed. A new Section 2.1 now explains both methods in the main text: what maximum entropy optimizes and under which constraints (Eq. 1), how we fit it, how the Gaussian copula is built and computed, and what we do when its correlation matrix is invalid (about 30% of eight-model groups).

W3 (selection benefit under realistic restrictions). We think the worry is that the gain comes mostly from picking weak but "different" models. Part of it does: if we only allow each pool's most accurate quarter of models, the gain at eight models goes from 3.3 to 2.0 percentage points. But it does not go away. We also added a stricter test where the chosen group can lose at most two points of average accuracy compared with the most accurate models. The gain is still 2.2 points, and the members' average accuracy on new items drops by only 1.7 points (Section 5, Table 15). So the benefit is smaller under these limits, but on average it remains.

W4 / Q1 (model properties). We checked whether the pattern depends on models sharing a family, a size, or a base model. It does not seem to. Groups with at most one model per family reconstruct about as well as random groups (16.2% vs 15.7% error at eight models, difference not significant). Groups of only small models, only large models, or only fine-tunes of one base model are all within about a point of random groups. Training data is not public, so we cannot measure training-data overlap directly; a shared base model is only a partial stand-in for it (Section 3, Appendix I).

AuthorsOpenReview ↗

Response to Reviewer 6gpy (2/2)

W5 / Q1 (item properties). Grouping items by difficulty accounts for about 80% of the extra overlap in failures between pairs of models. But difficulty alone does not reproduce collective failure: if models failed independently within difficulty levels, the estimate would be off by 44%, against 15.9% for maximum entropy. Maximum entropy also stays far closer than independence in every answer format, and its error is in a similar range across MMLU subjects. Items flagged as faulty by MMLU-Redux appear more often among collective failures. But removing them does not raise the reconstruction error more than removing other items matched on the number of collective failures (Appendix J).

W6 / Q3 (why pairs dominate, and when this should break). We agree the finding is descriptive, so we looked at a possible explanation from item response theory (Section 4). This model has exactly the pairwise maximum-entropy form under one condition: the hidden difficulty of the items that every model passes is normally distributed. That condition does not hold in general, even when difficulty itself is normal. In an exact calculation, maximum-entropy error at eight models is 3.5% when difficulty is normal, 28.5% when it is skewed, and 59.1% when it is bimodal (Table 11). The model also predicts that the error grows with group size, which is what we see (2.7% at three models, 15.9% at eight). So we expect the pattern to weaken for larger groups and for benchmarks with very uneven difficulty. This helps explain when the approximation can work and when it can fail, but it is not a general theory of language-model failures.

Q2 (estimators that use metadata). Difficulty-based estimators need extra item-level data (other models' results on each item), which a pairwise table does not contain. With that data they do help on new items (error 33.2% to 27.3% at eight models), but direct counting with the same difficulty information does just as well (28.0%). So the gain comes from the extra data, not from a better estimator. For choosing groups, family metadata alone barely helps (one model per family lowers collective failure by 3 to 4%, not significant), while choosing from the pairwise table does better (Section 5, Appendices G and J).

What changed in the revised PDF:

  • Abstract and introduction rewritten so the main terms are defined when introduced; Sections 2 and 3 simplified.
  • New Section 2.1 with the method details.
  • A new accuracy-limit test in Section 5 (Table 15).
  • New paragraphs in Section 3 (model families, item difficulty) and Section 4 (item response theory).
  • New Appendices I and J (model and item properties), and the hidden-difficulty model in Appendix F (Table 11).

Overall, the reconstruction result is unchanged, and the selection gain is smaller under realistic limits but does not disappear.

Reviewer_J8jcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the problem of estimating the collective failure rate of groups of language models by using individual and pairwise failure rates. They observe that maximum entropy can estimate or reconstruct values across multiple datasets. They use this information and show that it can be used to select models that have lower failure rates (all of them fail on an item) instead of choosing the models with the highest accuracy.

Critical strengths

The paper is well-written and well-organized. The problem is clear, and it is systematically evaluated with extensive experiments. Also, the authors have studied the limitations of their approach in held-out prediction, across different group sizes, and in model selection experiments.

Critical weaknesses

Do authors think that the high performance of pairwise reconstruction can be explained by the model pools that have variants of models from the same family? Have the authors evaluated whether the main reconstruction results still hold when restricting groups to one model per family or otherwise controlling for models within families?

Apart from model dependence, have the authors considered the inherent difficulty of the items and whether this can partially explain the high performance of reconstruction? For example, the case that most of the samples are either easy (all models are correct)or hard (all models fail).

I suggest including a brief explanation of the reconstruction methods in the main part of the paper, as it's critical to the conclusion.

Questions for authors

NA

1 comment
AuthorsOpenReview ↗

Response to Reviewer J8jc

Thank you for the kind review. All three points were good ones, so we ran the tests you suggested and put them in the paper. The changes are listed at the end.

W1 (models from the same family). Short answer: the reconstruction result does not seem to depend on including models from the same family. We ran exactly the test you describe: for every pool and group size we drew groups with at most one model per family and scored them like the paper's random groups. At eight models the error is 16.2% for one-per-family groups and 15.7% for random groups of the same pools, and the difference is not significant at any group size (Table 17). We also tried the opposite extreme. On the Open LLM Leaderboard, groups made only of fine-tunes of one base model fail together more often, but they are reconstructed just as well as random groups (Table 18). You also asked about other ways to control for related models. We excluded near-duplicate models (pairs that get almost the same items right and wrong) and tried several other controls. The overall reconstruction pattern remained similar (Table 17, Appendix I).

W2 (item difficulty). Partly yes, but it does not explain why the reconstruction works. First, the all-easy / all-hard case is not common. In a group of eight models, 69% of items are mixed (some members fail, some pass), and only 9.6% of a pool's items are passed by every model or failed by every model. If we remove those items, relative error goes up (16.1% to 21.5%). That is mostly what you'd expect, since collective failure gets rarer and relative error is larger for rare events: against untrimmed groups of the same pool with the same failure rate, the trimmed groups show no clear difference at four to eight models (Table 19). Second, we tested whether models fail together mainly because they encounter the same difficult questions. We estimated question difficulty using other models in the pool and divided questions into four difficulty levels. Grouping questions by difficulty accounts for about 80% of the extra overlap in failures between pairs of models. However, assuming that models fail independently within each difficulty level still gives 44.0% error when estimating collective failure for eight models, compared with 15.9% for maximum entropy (Table 20). Thus, question difficulty matters, but this simple explanation cannot fully account for how models fail together.

W3 (methods in the main text). Agreed, and done. A new Section 2.1 explains the compatible interval (a linear program), pairwise maximum entropy (its objective in Eq. 1 and how we fit it) and the Gaussian copula (how it is built and what we do when its correlation matrix is invalid). The other completions stay in Appendix D.

What changed in the revised PDF:

  • New Section 2.1 with the methods.
  • A new paragraph in Section 3 on model families and item difficulty, also mentioned in the conclusion.
  • New Appendix I: one model per family, six controls, model size and shared base model (Tables 17 and 18).
  • New Appendix J: items every model passes or fails, difficulty levels, held-out results, answer format, subject and faulty items (Tables 19 to 23).
  • The supplement also has a new list of each pool's model families.

Both checks leave the main result in place: maximum entropy stays close to the observed rate with one model per family, and item difficulty alone does not explain it.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision: what changed, why, and what stayed the same

Thank you to all three reviewers for the detailed feedback. We have revised the paper and run additional experiments to address your concerns. We want to clarify what has changed and, importantly, what has not.

The central question and findings remain unchanged. We still ask how much of a model group's collective failure can be recovered from its individual and pairwise failure rates. The original 48 evaluation pools, main estimators, and primary experiments remain the same. So do the headline results: at eight models, the pairwise table allows an interval averaging 16 percentage points in width, yet maximum entropy reconstructs collective failure with 15.9% relative error (1.1 percentage points). Groups selected using the pairwise table also fail together on held-out items 3.3 percentage points less often than groups of the most accurate models. The main result tables are unchanged apart from their captions.

Why we ran the additional experiments. The reviews raised important questions about whether these findings depend on particular features of the data or hold under more realistic conditions. We tested those possibilities directly.

  • Model families and item difficulty (J8jc, 6gpy): We tested whether reconstruction works mainly because models are closely related or because some questions are easy or hard for everyone. The results remain similar when models come from different families or share a size or base model, while difficulty alone does not reproduce collective failure. We also checked answer format, subject, and faulty items (Section 3, Appendices I–J).
  • Higher-order dependence (6gpy, pDgx): We extended the interaction-order analysis to every pool with enough items, including open-ended and agentic tasks at smaller group sizes. Pairwise terms still account for most measured dependence. We also added an explanation from item response theory, with a latent-difficulty model that shows when the approximation should weaken, and clarified why the eight-model analysis remains limited (Section 4, Appendix F).
  • Realistic group selection (6gpy, pDgx): The original selection gains might partly come from choosing weak models. We therefore limited how much average member accuracy could be sacrificed. At a two-point accuracy budget, the held-out collective-failure reduction remains 2.2 percentage points (Section 5, Appendix G). We also cut the candidates to a fixed number of the most accurate models; the gain grows as more candidates are allowed (Appendix G).
  • Prediction limits (pDgx): We added a comparison with direct counting when only a few items are available for fitting (Appendix K), and we state the limits for incomplete records and cross-benchmark transfer more clearly (Sections 3 and 7, Appendix L). These clarify where pairwise estimation is useful and where it is not.
  • A data caveat we added ourselves: In one release (EmbedLLM), prompt variants of the same GPQA question can fall in both the fit half and the held-out half. We now state this and report held-out and selection results without the ten EmbedLLM pools (Appendices D, G and K). Group selection still helps without them (2.6 percentage points at eight models), while held-out error is higher (38.5% against 32.4%).

We also improved the presentation. The abstract, introduction and Sections 2 and 3 are rewritten in plainer language, with the main terms defined when introduced. Section 2.1 now explains the estimators. Section 5 defines the strong-quarter rule and moves the details of the two further collections to Appendix G. The Discussion (Section 7) separates the mathematical result from the empirical finding. Appendix B and the figure and table captions state which pools each test uses. Repeated text in the appendix was shortened, and four references were added for the new analyses. Most new results are in the appendix, with short explanations in the main text.

In short, the additional experiments test the assumptions behind our original findings. They do not overturn those findings, but they make their scope and limitations clearer. We have not introduced a new central method or changed the paper's research question. The revised main text remains within the 10-page limit, and the supplementary material includes the new analyses and reproduction code.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.