- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper is extremely difficult to understand, as detailed later. The following is a best-effort guess of what the paper is about.
The paper studies between-datapoint dependencies in the scores of multiple evaluated models and how they affect the effective sample size of the evaluation data. Dependencies between evaluation datapoints can make the effective sample size smaller than the nominal size, but only if the dependencies affect all models equally. Dependencies that affect the order between models are useful signal, so they do not reduce effective sample size.
Critical strengths
None
Critical weaknesses
The paper is extremely difficult to understand, to the point that it is not fully clear what is studied and how the theory and experiments relate to that. It appears that the effective evaluation width (which seems like effective sample size) is the main target of study. It is defined in Eq. (2), and again in Eq. (9), but the text does not explain or motivate the definition at all. Section 3.1 also defines a bunch of other quantities that are not explained or motivated at all. The experiments follow the same patter: the figures are not explained in the text, and Figures 2 and 4 are not even referenced. The text appears to discuss completely different results, but does not explain how they relate to the rest of the paper.
A large component of the difficulty is that the paper is full of non-standard jargon that is not explained. Examples include: "evaluation unit", "evaluation width", "marginal participation ration", "effective rank", "spectral diversity", "candidate-centered score geometry" just in the first two paragraphs of the Introduction.
Other points:
- The reproducibility statement claims that "supplement documents the comparison-width identities, preprocessing, aggregation levels, synthetic interventions, and held-out evaluation", but there is no supplementary material of any kind.
- The datasets are proprietary according to the reproducibility statement, so reproducing the results is not possible.
Questions for authors
None
2 comments
Thank you for the response. The updated paper has added some explanations, but these are just as hard to understand as the rest of the paper and do not clarify what the studied quantities are. As a results I'm keeping my score.
Reply to 33Rk
Thank you for the detailed diagnosis; we accept the presentation problems.
What is studied. Two patterns matter over the same evaluation units. First, candidate models disagree more on some units than on others; this comes from their scores. Second, some evaluation units move together; this comes from past returns. We ask whether these two patterns line up. If the models disagree mainly on units that move together, one shared fluctuation can move much of the comparison in one direction; if the models agree on those units and disagree elsewhere, the same dependence matters much less. The general response gives a bank-sector example.
Clarifying a key point our original text obscured. A component shared by all candidates cannot change their ordering, so we remove that common component first. The dependence that matters is the dependence aligned with candidate disagreement, because that shared movement can change which candidate looks better.
How theory and experiments connect. §3 defines the comparison-specific overlap and reference width. §4 measures them on real stock panels before evaluation outcomes are used. §5 changes the dependence structure in controlled synthetic experiments and measures how comparison noise responds. Appendix T then directly compares predicted and observed comparison noise on real outcomes.
Eq. 2, Eq. 9 and figures. You are right that the original text did not motivate these objects or connect the figures clearly. The revised Introduction now starts from the concrete bank-sector example before introducing the geometry; §3.1 derives Eq. 9 in one line and adds a notation table with intuition, and the text explicitly walks through Figures 1, 2 and 4.
Supplement and reproducibility. We agree that the supplementary material was not sufficiently visible or usable. It is now merged into the revised PDF as appendices. Because the real market data are licensed and cannot be redistributed, we have also uploaded an anonymous synthetic-only reproduction package as supplementary material. The package contains no real market data, and its README maps the included scripts and artifacts to the synthetic results in the paper.
We have uploaded a revised PDF; major additions and corrections are marked in blue, and the former supplementary material is now included as appendices in the same PDF for ease of review.