- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether pairwise co-failure statistics of language models can estimate the collective all-failure rate, the probability that every model in a group fails on one input. Theoretically, pairwise statistics only constrain this rate within an interval and cannot uniquely determine it. The authors find that the maximum-entropy method produces strong approximations, and that for the group sizes studied, most of the measured failure dependence is captured by pairwise terms.
Critical strengths
The empirical scope is broad, spanning substantial model pools and benchmarks, and the findings are connected to a principled maximum-entropy framework.
Critical weaknesses
(1) The motivation is unclear. The estimation in the main experiments seems to require complete test results, which are then compressed into a pairwise table and used to reconstruct the joint failure probability. If the complete results are already available, it is unclear why this extra step is needed. Although the paper also studies incomplete logs, the estimation errors appear to remain substantial, leaving the practical benefits insufficiently established. In addition, the paper’s own experiments also show that the estimate does not transfer well across benchmarks. The method performs substantially worse in held-out estimation than in reconstruction, and since it relies on pool-specific pairwise error statistics, its ability to transfer across benchmarks or datasets appears limited and may require additional target-distribution measurements—limiting its practical applicability when such measurements are costly or unavailable.
(2) The “mostly pairwise dependence” explanation is only partially validated: the interaction-order analysis covers 13 pools at k=6 and only 3 MMLU-family pools at k=8, leaving higher-order effects insufficiently characterized on open-ended, code, and agentic tasks.
Questions for authors
Please see weaknesses.
4 comments
Response to Reviewer pDgx (2/2)
Weakness (2). Thanks, this is a fair point. The analysis was narrow because it needs a lot of items. Measuring every order of dependence among k models requires enough items to estimate the probabilities of all 2^k failure patterns reliably, so we kept the item rule from the submitted paper: at least 40 × 2^k items (2,560 at six models, 10,240 at eight). At eight models only three pools, all MMLU-type, are that big.
So we ran the analysis on every pool the rule allows at smaller group sizes: 36, 28 and 22 pools at three, four and five models. Pairs still carry about 96 to 97% of the dependence on average, and no single pool goes below about 90% (Section 4, Table 10). This includes open-ended tasks: open-ended math and other open-ended generation both stay above 94% (Table 22). For agentic coding, SWE-bench Verified and SWE-bench Pro are only big enough for three models, and there pairs carry about 99% (Appendix F).
Code-generation pools are too small for this analysis at any size (50 to 78 items), and the agentic pools are too small beyond three models. There we checked the thing the analysis is meant to explain, collective failure itself. At eight models, maximum entropy has 8.3% reconstruction error on code generation and 8.0% on SWE-bench Verified. On SWE-bench, the range the pairwise table alone allows is far wider: its half-width is nearly six times that error (Appendix H, Table 22). These results extend the reconstruction evidence to coding tasks, although we still cannot measure the full eight-model breakdown by interaction order on them.
We agree the result is descriptive and limited to the group sizes the data supports, and the paper says so (Section 4).
What changed in the revised PDF:
- Section 4 now states the item rule in the main text and extends the analysis to every pool it allows, including open-ended tasks and agentic coding.
- New Table 10 (Appendix F) with the shares on all these pools.
- New Table 22 with results by answer format.
Across both comments: pairs carry most of the measured dependence on every pool the data allows, and the limits you raise, held-out error and no transfer across benchmarks, are stated in the paper as limits.
Thank you for the detailed response. The results demonstrate that pairwise statistics preserve substantial information about collective failure, and the additional group-selection experiments help clarify this finding. However, the motivation and practical value are still unclear to me. First, the authors mention that benchmark owners could release pairwise statistics instead of item-level results, but it is unclear how common this scenario is or how useful the method would be in practice. Even if only these pairwise statistics are available, they may have been collected under inconsistent evaluation conditions—for example, different decoding settings for the same model across pairs. Given that these conditions may affect the failure statistics, how can such summaries be correctly combined into a joint model. Thus, I maintain my original score.
Reply to pDgx follow-up: motivation, practical value and evaluation conditions
Thank you for following up. We think we could have explained the motivation of our work more clearly.
Our main contribution is an empirical finding about how language models fail together, rather than a new estimation method. Suppose a future study develops a method based on the assumption that pairwise failure statistics preserve substantial information about larger groups. That assumption would itself need substantial empirical evidence. Providing that evidence is the purpose of our paper. We use maximum entropy as a tool to study this question across 48 evaluation pools and 13,877 model groups. The experiments on model families, item difficulty, interaction orders, and held-out prediction examine how robust this finding is and where it breaks down.
Regarding practical value, there are already studies that release pairwise statistics. For instance, Kim et al. (2025) publish individual model accuracies and pairwise outcome statistics for HELM and Hugging Face models (https://github.com/nikhgarg/llm_correlated_errors_public).
Consider a benchmark such as FrontierMath. Its questions are unpublished, which reduces the chance that models have seen them in training, and model traces can be viewed only for the 10 public questions (https://epoch.ai/benchmarks/about). When only each model's score is available for the private questions, researchers can tell which models perform better, but not whether two models fail on the same questions or complement each other. Pairwise failure statistics would fill this gap. The benchmark owner can compute them from the private results and release them without revealing any question. The community could then study many different combinations of the evaluated models without access to the per-question results.
We are not claiming that this is already common practice. Rather, we encourage benchmark owners to release these statistics alongside individual accuracies, and our paper provides empirical evidence for why such a release could be valuable. One use is choosing groups of models that collectively solve more questions. The finding itself rests on the reconstruction results (Section 3); Section 5 instead tests whether it is useful in practice: the selection algorithm only uses pairwise statistics, and we retain the full response matrix as ground truth to evaluate the selected groups on held-out items. This is a measured result, not a projection: at eight models, the selected groups fail together 3.3 percentage points less often on held-out items than the most accurate models, and in a separate test about as often as groups selected from the full response matrix (10.03% against 9.95%, Appendix G).
We focused the main paper on establishing the empirical finding and examining its robustness and limitations. This is why we discussed the potential use of pairwise releases primarily in Section 7. From this discussion, however, we realize that this motivation deserves more attention. Specifically, we will explain the motivation for releasing pairwise statistics in Section 1, clarify how the coverage experiment in Section 5 simulates this setting, and expand Section 7 to discuss what such releases would enable. Given the limited time remaining in the discussion period, we plan to incorporate these clarifications in the camera-ready version.
Regarding inconsistent evaluation conditions, each evaluation pool in our study consists of multiple models evaluated on the same set of benchmark items. To illustrate the consistency issue with a simple example, suppose we let Model A answer question 1, change its decoding settings before it answers question 2, and continue doing so across the dataset. We would then be mixing results from different configurations of Model A in a single response vector. A similar problem arises in your example of using different configurations of Model A for different pairs.
Our setting is different. We start from the evaluation results of Model A under a given configuration across the dataset. We use this same response vector to compute its individual failure rate, its co-failure rate with Model B, its co-failure rate with Model C, and so on. In other words, the pairwise statistics are computed from a common response matrix after evaluation, rather than obtained by evaluating each pair separately. Different decoding configurations should be treated as separate systems. We agree that combining pairwise statistics from inconsistent evaluation conditions would be problematic, but that is not the setting we study.
We appreciate this discussion, as it helped us identify where the motivation of our paper needs to be communicated more clearly. Our main aim remains to establish an empirical finding about collective language-model behavior, with the coverage experiment demonstrating one use of that finding.
Response to Reviewer pDgx (1/2)
Thank you for this comment. We reply to weakness (1) here and to weakness (2) in a second comment.
We agree that when the complete response matrix is available, collective failure can be counted directly. We are not proposing maximum entropy as a replacement for that count. Our question is different: how much information about collective failure is lost when model evaluations are summarized by individual and pairwise failure rates?
This matters because correlated errors across language models are already studied and reported using pairwise statistics (Kim et al., 2025; Goel et al., 2025). We use complete response matrices as ground truth to test how informative such summaries are. At eight models, the pairwise table allows a range of collective failure rates averaging 15.8 percentage points in width. Yet maximum entropy reconstructs the observed rate with only 1.1 percentage points of mean absolute error (15.9% relative error; Section 3). The point is not that maximum entropy beats direct counting, but that much of the information about collective failure remains in the pairs.
We also test whether that information is useful for choosing model groups. At eight models, selecting groups using pairwise statistics reduces held-out collective failure by 3.3 percentage points compared with choosing the most accurate models. The reduction is 2.0 points when selection is restricted to strong candidates (44 pools), and 2.2 points when the selected group's mean accuracy can fall by at most two points on the fit items (Section 5; Appendix B lists the pools).
For group selection, the full response matrix gives only a small additional benefit. In a separate test where every method chooses from the same set of up to 300 candidate groups, the group chosen from the pairwise table fails together on 10.03% of held-out items at eight models, the group chosen from the full response matrix on 9.95%, and the most accurate candidate group on 11.58%. The difference between pairwise selection and direct counting is not statistically resolved (Appendix G). So the pairs keep almost all of the benefit of the full data. On SWE-bench Verified, groups chosen from pairs also fail together on 9–15% fewer held-out tasks than the most accurate systems, though not at eight systems in its one-agent subset (Section 5, Appendix H).
Prediction on new items is harder, as you note. At eight models, maximum entropy misses by 32.4%. Direct counting on the full fit-half data misses by almost as much (30.3%). So even the full data does not fix this, and both errors drop sharply when the fit half has more collective failures (Section 3, Figure 3). With incomplete records, pairs do better than counting on the few items every model shares, but scoring all models on one common set of items is better when that is possible (Section 3, Appendix L).
We agree that pairwise statistics do not reliably transfer across benchmarks (Section 7, Appendix L), and we do not claim otherwise. The value we demonstrate is narrower: within an evaluation, pairwise statistics retain enough information to select useful model groups without access to the full response matrix.
One possible use is for benchmark owners to publish pairwise statistics without releasing item-level responses (Section 7). A single such table could then be used to study many possible model groups. We have tested what remains possible under this information restriction, but have not demonstrated an actual deployment with pairwise-only access or a reduction in evaluation costs.
So the paper measures what pairwise statistics keep and where they fall short: the selection results on new items show that what they keep is useful, and prediction on new items, incomplete records and transfer show its limits.
What we changed in the revised PDF for this comment