- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes Inferred Generative-Process diversity (IGP), which is measured from open-ended model responses using compression distances and a byte-permutation control. It tests whether this measure relates to shared wrong answers on separate benchmarks. Experimental results show an association beyond the tested semantic baseline and model capability.
Critical strengths
-
Understanding when models make the same mistakes is important because shared errors undermine systems that rely on multiple models. The paper studies a problem relevant to improving reliability.
-
The paper measures diversity and shared errors on separate response datasets, which helps test whether the relationship holds across tasks. Controlling for capability and semantic distance also helps assess whether IGP provides additional information.
Critical weaknesses
Majors:
-
The task-transfer design shows that IGP measured from open-ended responses is related to shared errors on separate question-answering tasks (p.2, L75-081, p.7, L351-362). However, this does not yet demonstrate that selecting a team with high IGP makes a system more reliable, the reported ensemble comparison measures chance-corrected convergent failure (p.5, L246-256, L264-269; p.8, Fig.4). Testing a fixed team with a clear decision rule (such as voting or abstention) on new tasks would strengthen this claim.
-
The evaluation metric changes between the two-model and five-model settings. Pairwise CWA conditions on cases where at least one model is wrong (p.4, L201-210, Eq.3), whereas five-model CFR conditions on cases where all models are wrong (p.5, L246-256, Eq.5). This difference makes the two settings less directly comparable. Need clarification why different conditioning events are needed and test whether the conclusions hold when both settings use the same conditioning rule.
-
The reported effects are modest, with a mean partial correlation of after controlling for semantic distance and capability (p.7, L351-355). In the five-model experiment with semantic distance held fixed, only two of the five bands have 95% confidence intervals that exclude zero in the cross-benchmark summary (p.8, L401-414, Fig.4b), encouraging but the size and consistency of the benefit need stronger support.
Minors:
-
Notation like , (p.3, L123-127), and the Vendi score (p.5, L260-264), needs more explanation when first introduced in the main text. There could be more cases like this in the paper.
-
The English writing needs improvement, as parts of the paper are difficult to follow.
Questions for authors
- What is the evidence that supports the step from lower shared-error metrics to better performance under a team decision rule (p.5, L246-269, p.8, Fig.4)? A fixed-team test with voting or abstention would be one useful way to examine this.
- Why does the conditioning event change between pairwise CWA and five-model CFR, do the findings hold when both settings use the same conditioning rule?