acceptodds
Under review as a conference paper at ICLR 2027

Beyond Nominal Width: Comparison-Noise Geometry of Model Validation under Cross-Sectional Dependence

Abstract

Automated model discovery makes candidate generation cheap, but comparing candidates requires more than a large validation set. Counting evaluation units ignores both their dependence and how shared fluctuations affect differences between candidates. We introduce comparison effective width to characterize this comparison-noise geometry. The measure pairs candidate-centered score geometry with dependence among evaluation units. A projection-sufficiency result establishes that comparison rules invariant to common additive shifts depend only on centered candidate evaluation scores. Exact identities link comparison effective width to a comparison design effect and a second-order noise descriptor, all computable before reading evaluation outcomes. In label-free measurements across 2,258 matched cells from CSI300, a large-cap Chinese equity universe, as nominal width grows from 30 to 120, comparison effective width relative to nominal width falls from 0.726 to 0.389. The cell-level per-added-asset slope declines in 2,258/2,258 matched cells. CSI500, a disjoint universe in the same market, shows the same descriptive pattern with weaker attenuation. A same-candidate decomposition locates the CSI500–CSI300 gap in log comparison effective width mainly in return-side dimensionality and score–return alignment. Controlled synthetic experiments further show that the eigenvalue spectrum of the full covariance matrix alone does not determine split-half ranking agreement. An empirical relation combining cross-half candidate-IC covariance with zero-signal split-half IC-difference dispersion predicts this agreement on held-out synthetic worlds with cross-fitted R² ≈ 0.88 and systematic calibration deviations. Together, these results show why nominal evaluation counts should be complemented by a measure that accounts for evaluation-unit dependence and candidate-score geometry.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Model evaluation, Model selection, Effective dimension, Cross-sectional dependence, Comparison noise

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_33RkOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper is extremely difficult to understand, as detailed later. The following is a best-effort guess of what the paper is about.

The paper studies between-datapoint dependencies in the scores of multiple evaluated models and how they affect the effective sample size of the evaluation data. Dependencies between evaluation datapoints can make the effective sample size smaller than the nominal size, but only if the dependencies affect all models equally. Dependencies that affect the order between models are useful signal, so they do not reduce effective sample size.

Critical strengths

None

Critical weaknesses

The paper is extremely difficult to understand, to the point that it is not fully clear what is studied and how the theory and experiments relate to that. It appears that the effective evaluation width neffcn_{eff}^c (which seems like effective sample size) is the main target of study. It is defined in Eq. (2), and again in Eq. (9), but the text does not explain or motivate the definition at all. Section 3.1 also defines a bunch of other quantities that are not explained or motivated at all. The experiments follow the same patter: the figures are not explained in the text, and Figures 2 and 4 are not even referenced. The text appears to discuss completely different results, but does not explain how they relate to the rest of the paper.

A large component of the difficulty is that the paper is full of non-standard jargon that is not explained. Examples include: "evaluation unit", "evaluation width", "marginal participation ration", "effective rank", "spectral diversity", "candidate-centered score geometry" just in the first two paragraphs of the Introduction.

Other points:

  • The reproducibility statement claims that "supplement documents the comparison-width identities, preprocessing, aggregation levels, synthetic interventions, and held-out evaluation", but there is no supplementary material of any kind.
  • The datasets are proprietary according to the reproducibility statement, so reproducing the results is not possible.
Questions for authors

None

2 comments
AuthorsOpenReview ↗

Reply to 33Rk

Thank you for the detailed diagnosis; we accept the presentation problems.

What is studied. Two patterns matter over the same evaluation units. First, candidate models disagree more on some units than on others; this comes from their scores. Second, some evaluation units move together; this comes from past returns. We ask whether these two patterns line up. If the models disagree mainly on units that move together, one shared fluctuation can move much of the comparison in one direction; if the models agree on those units and disagree elsewhere, the same dependence matters much less. The general response gives a bank-sector example.

Clarifying a key point our original text obscured. A component shared by all candidates cannot change their ordering, so we remove that common component first. The dependence that matters is the dependence aligned with candidate disagreement, because that shared movement can change which candidate looks better.

How theory and experiments connect. §3 defines the comparison-specific overlap and reference width. §4 measures them on real stock panels before evaluation outcomes are used. §5 changes the dependence structure in controlled synthetic experiments and measures how comparison noise responds. Appendix T then directly compares predicted and observed comparison noise on real outcomes.

Eq. 2, Eq. 9 and figures. You are right that the original text did not motivate these objects or connect the figures clearly. The revised Introduction now starts from the concrete bank-sector example before introducing the geometry; §3.1 derives Eq. 9 in one line and adds a notation table with intuition, and the text explicitly walks through Figures 1, 2 and 4.

Supplement and reproducibility. We agree that the supplementary material was not sufficiently visible or usable. It is now merged into the revised PDF as appendices. Because the real market data are licensed and cannot be redistributed, we have also uploaded an anonymous synthetic-only reproduction package as supplementary material. The package contains no real market data, and its README maps the included scripts and artifacts to the synthetic results in the paper.

We have uploaded a revised PDF; major additions and corrections are marked in blue, and the former supplementary material is now included as appendices in the same PDF for ease of review.

Thank you for the response. The updated paper has added some explanations, but these are just as hard to understand as the rest of the paper and do not clarify what the studied quantities are. As a results I'm keeping my score.

Reviewer_AyfbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1(−1)
Read review
Paper summary

This paper studies how dependence among evaluation samples affects the comparison of candidate models. The authors propose comparison effective width, which considers both sample dependence and differences between candidates. They derive several mathematical properties and conduct experiments on CSI300, CSI500, ETF data, and synthetic datasets.

My main concerns are: (i) the proposed noise measure does not always agree with the actual RankIC comparison (W1); (ii) the novelty relative to closely related methods remains unclear (W2); and (iii) the experiments do not sufficiently show that the measure reflects actual comparison uncertainty (W3).

Critical strengths
  • Relevant problem. The paper addresses an important issue: having more evaluation samples does not necessarily mean that candidate models can be compared more reliably, especially when samples are correlated.
  • Experimental observations. The experiments show consistent patterns across CSI300 and CSI500, and the synthetic experiments demonstrate that covariance orientation matters even when the overall spectrum is unchanged.
Critical weaknesses

W1. Inconsistent Comparison

Scores are normalized on the full asset set and then restricted without re-ranking or re-centering. However, the actual Spearman RankIC uses ranks computed within each evaluation subset.

This difference can lead to incorrect noise measurements. For instance, two candidates can have identical rankings within an evaluation subset, making their RankIC difference always zero, while the proposed DgeomD_{\mathrm{geom}} in Eq. (4) remains positive.

I concern about whether the measure captures the noise of the actual comparison statistic. The authors should use consistent preprocessing and verify whether the main results still hold.

W2. Related works

The proposed effective width is closely related to existing methods, particularly [R1].

  • R1 already combines effective dimensionality and covariance alignment. In fact, using the same matrices, the proposed width can be expressed directly in terms of ENSD: neffc=PR⁡(Csc)PR⁡(Cr)ENSD⁡(Csc,Cr).n_{\mathrm{eff}}^c=\frac{\operatorname{PR}(C_s^c)\operatorname{PR}(C_r)}{\operatorname{ENSD}(C_s^c,C_r)}.

  • R2 provides the classical foundation for adjusting effective sample size using dependence structures.

The main difference appears to be the candidate-centering operation and its interpretation for model comparison. While this is reasonable, it remains unclear whether it provides meaningful benefits beyond existing measures.

I suggest directly comparing the proposed method with these approaches and clarifying what is genuinely new.

W3. Insufficient Evidence

  • The real-data experiments examine the proposed measure itself, but do not show whether it predicts actual comparison errors. The experiments also rely mainly on the same Alpha158 candidate family within one market.
  • Direct reliability prediction from the proposed geometry performs poorly (R2=−1.316R^2=-1.316). The later result (R2≈0.88R^2\approx0.88) instead uses statistics estimated from synthetic outcomes, so it does not directly validate the proposed label-free measure.
  • The planned real-outcome validation was not completed because the estimator-recovery check failed under zero injected effect in all 256 worlds of each scenario.

References

[R1] The Effective Number of Shared Dimensions Between Paired Datasets. Giaffar et al., AISTATS, 2024.

[R2] Modifying the t Test for Assessing the Correlation Between Two Spatial Processes. Dutilleul, Biometrics, 1993.

Questions for authors
  1. The authors use a 250-day historical window without covariance regularization. How sensitive are the results to the window length and covariance estimator?

  2. The interventions preserve the full covariance spectrum, but not the variances or spectra of the two evaluation halves. Could these changes also explain the observed differences in reliability?

  3. The prediction errors vary across widths and signal strengths. What causes these differences, and would more synthetic repetitions improve calibration?

2 comments
AuthorsOpenReview ↗

Reply to Ayfb, Q1–Q3

Q1 (covariance estimation). We found an error; the main conclusions survive it. CSI300 used daily cross-sectional demeaning, while CSI500 first removed an equal-weight market factor (assetwise OLS with intercept); the paper describes both as demeaning. Recomputing with demeaning in both leaves CSI300 unchanged and lowers CSI500 widths slightly. At the corrected 250-day baseline, three conclusions hold: the gain per added stock still shrinks in every matched cell, CSI500 remains wider than CSI300 on average, and the three components of the gap keep the same order. One Figure 3 statement narrows: the participation ratio of the return correlation matrix divided by the number of stocks, PR(C_r)/n, is higher for CSI500 at 18 of 21 origins, not at every origin. The ETF–CSI300 comparison in §4.3 uses demeaning on both sides and is unaffected.

Sensitivity check (completed). We repeated the analysis in eight settings: return windows of 60, 120, 180 and 250 days, each with the plain sample correlation and with Ledoit–Wolf shrinkage, without reading evaluation outcomes.

  • Holds in all eight settings: the gain per added stock shrinks in every matched cell, and CSI500 is wider than CSI300 on average at every width.
  • Depends on the setting (a): which component of the gap is largest. The return-dependence component is the largest at the 250-day baseline and in every shrinkage setting. With 60- or 120-day sample correlations, the alignment component is larger at 120 stocks.
  • Depends on the setting (b): the Figure 3 pattern. With sample correlations, PR(C_r)/n is higher for CSI500 at 18 of 21 origins with 250 days, but at only 11 of 21 with 60 days. A 60-day sample correlation has rank at most 59, which limits this ratio more for the larger universe; the weakening is consistent with that limit, though we have not shown it is the only reason. In a follow-up check that subsamples CSI500 to the CSI300 number of stocks at each origin, so that both face the same limit, PR(C_r)/n is higher for CSI500 at 21 of 21 origins for all four windows. With shrinkage the count is 19 to 21 of 21, but shrinkage is stronger for CSI500 and itself raises this ratio, so we do not count it as extra evidence.

We therefore present (a) and (b) as results at the 250-day baseline, not as general ones.

Revision: corrected values now appear in §4, the revised main-results table, Figures 2–3 and the corresponding appendix sections; submitted values and sensitivity checks are retained in an added appendix.

Q2 (synthetic interventions). Yes. The intervention changes more than orientation, so it does not isolate orientation as the sole cause. Although the full-population spectrum is fixed, basic properties of the two halves also change: their traces by up to 29% and their participation ratios by up to 107% (Appendix H). The supported conclusion is that full-population spectral summaries are insufficient. Revision: §5.1 now uses the corrected wording and reports the half-level changes.

Q3 (prediction calibration). The calibration differences are structured rather than primarily Monte-Carlo noise. In the complete 3×23\times2 width-by-SNR diagnostic, residuals become more negative as predicted reliability increases within a stratum, while a separate width-associated offset remains across strata; the weakest condition is width 120 / SNR 30 (R2=0.770R^2=0.770, slope 0.7830.783). We therefore do not attribute the variation to a single scalar cause, but it is clearly not explained by random repetition noise alone.

The replicate comparison helps answer whether simply running more simulations would fix it. When predictor and target reuse the same replicates, the same-half and full-sample fits give (R^2=0.9324) and (0.9313); using disjoint replicate folds lowers this to (0.8814). Thus removing predictor–target coupling matters more than merely halving the number of replicates. More repetitions may reduce Monte-Carlo error, but they would not by themselves remove the structured width/SNR calibration pattern.

Revision: §5.3 now reports the full width-by-SNR residual diagnostic and distinguishes sampling variation from the remaining structured calibration error.

We have uploaded a revised PDF; major additions and corrections are marked in blue, and the former supplementary material is now included as appendices in the same PDF for ease of review.

Thank you for the comprehensive rebuttal and substantial revisions. I appreciate the additional real-outcome experiments, which meaningfully address W3, as well as the robustness analyses that largely resolve Q1.

However, I still have two main concerns. First, while the empirical impact of the estimand mismatch (W1) appears limited, the original geometric descriptor remains different from the actual subset-local RankIC statistic, and the real-outcome validation relies on a modified predictor. Second, regarding W2, the novelty beyond existing ENSD-based approaches remains somewhat limited, particularly given the small empirical impact of candidate centering.

The new results provide encouraging evidence for predicting split-half comparison noise, but do not directly establish reliability in terms of actual model-ranking errors, and validation remains limited to one candidate family.

Overall, I appreciate the improvements, but will maintain my current rating.

Reviewer_jb85OpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how reliably one can evaluate models if evaluation units are correlated or dependent in some way. The central observation is that the nominal number of evaluation samples may overstate the amount of independent information available for comparing candidate models, particularly when the samples are correlated. The authors argue that the relevant dependence structure is not determined by the covariance of evaluation samples alone, but also by its alignment with differences in candidate predictions. To characterize this, they introduce comparison effective width, a quantity that combines candidate-centered score geometry with evaluation-unit covariance. They derive algebraic relationships connecting this measure to existing notions of effective dimensionality and design effects, and establish that comparison rules invariant to common additive shifts depend only on candidate-centered statistics.

The authors then run an experiment using stock data, along with controlled synthetic experiments. The real-data experiments show that comparison effective width increases sublinearly with the number of evaluated stocks, with stronger attenuation in CSI300 than CSI500. A decomposition attributes the difference primarily to return-side dimensionality and covariance alignment. Synthetic experiments further demonstrate that covariance matrices with identical eigenvalue spectra can induce different split-half ranking reliability depending on their orientation relative to candidate scores. The authors also present an empirical signal/noise relation that predicts split-half ranking reliability on held-out synthetic experiments with an (R^2) of approximately 0.88. However, the real-data experiments are conducted without future evaluation outcomes, and the paper does not establish whether comparison effective width predicts realized model-ranking errors.

Critical strengths
  1. The paper addresses an interesting problem. Nominal evaluation-set size can be misleading when evaluation units are correlated.
  2. The theoretical definitions and results are well defined and the proofs appear correct to me.
Critical weaknesses

My primary issue is that the presentation of the paper makes it incredibly difficult to parse the contributions and understand the results, despite my best efforts to read the paper. I would encourage the author to describe the paper in more detail in their own words during the rebuttal. After that, I'm happy to give a more detailed review.

Questions for authors
  1. Can you provide a practical interpretation of comparison effective width?
2 comments
AuthorsOpenReview ↗

Reply to jb85

Thank you for checking the definitions and proofs, and for asking for a practical interpretation. The bank example in our general response now also opens the revised Introduction. Here is how neffcn_{\mathrm{eff}}^c turns it into a number.

What the number means. In CSI300 with 120 stocks, neffcn_{\mathrm{eff}}^c averages about 46.6. Under our second-order description, comparing the candidates on these 120 correlated stocks is about as noisy as comparing them on 46.6 independent stocks with the same total variance. The value 46.6 is a reference width, not a literal count of independent stocks.

How it grows. Going from 30 to 120 stocks multiplies the number of stocks by four, but raises neffcn_{\mathrm{eff}}^c only from 21.8 to 46.6. Each added stock adds about 0.40 between 30 and 60 stocks, and about 0.22 between 60 and 120. Correlated stocks keep exposing the same candidate disagreement, so the comparison-specific width grows much more slowly than the number of stocks.

Revision. The revised §3.1 now gives this reading right after the definition. We would welcome your more detailed review. We have uploaded a revised PDF; major additions and corrections are marked in blue, and the former supplementary material is now included as appendices in the same PDF for ease of review.

Thank you for the reply, but I still find the paper incredibly difficult to understand. This stocks example makes sense but once the paper starts discussing the technical details, the nonstandard terminology and writing style make it difficult to follow and interpret any of the results. Thus, I will keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers. All three found the paper hard to follow, and we agree: we did not explain the basic question clearly. We start with an example.

The problem, in one example. Two candidate models, A and B, each rank the same 120 stocks by predicted future return. Forty of the stocks are banks, and banks tend to rise and fall together. Suppose A and B disagree mainly on the banks. If banks rise or fall together, that one sector move can shift much of the A-versus-B comparison at once. That is largely one shared shock seen through many stocks, not the same amount of independent comparison evidence as 40 unrelated observations.

Now keep the same 120 stocks and the same bank co-movement, but compare two other models, C and D, that rank the banks almost identically and disagree elsewhere. The dependence is the same, but the comparison is much less affected by the bank shock.

The question. The evaluation units and their dependence can therefore stay fixed while the noise of a model comparison changes with the pair of models. We ask not only how dependent the evaluation units are, but whether that dependence falls where the candidates disagree.

What we add. We combine two ingredients: where the candidates disagree (from their scores) and which evaluation units move together (from past returns). Their overlap defines neffcn_{\mathrm{eff}}^c, a comparison-specific reference width. Under our second-order description, it is the width of an independent reference evaluation set, with the same total variance, that would show the same comparison-noise level. It is not a literal count of independent stocks. The formula itself is not new algebra (reply to Ayfb, W2); the contribution is the comparison-specific choice of the candidate-disagreement object paired with evaluation dependence.

Evidence. In controlled synthetic experiments, cases with the same full-population dependence spectrum can have different comparison noise when dependence is arranged differently relative to candidate disagreement. In response to W3, we also added direct real-outcome tests. On primary CSI500 in the later 2021–2025 period, the median within-origin Spearman correlations between predicted and observed comparison noise are 0.36, 0.49 and 0.38 at widths 30, 60 and 120, with 19/20 origins positive at every width. Candidate-blind dependence baselines are lower at all three widths.

A correction and robustness result. We found one preprocessing inconsistency between CSI300 and CSI500 and corrected it. The two main width patterns survive the correction and all eight window-length/covariance-estimation settings we tested: the marginal effective-width gain from adding stocks declines, and CSI500 has larger average comparison effective width than CSI300. Thus adding more stocks still adds comparison evidence, but not one-for-one with nominal width: the marginal gain diminishes.

Revision. The revised Introduction now opens with this bank-sector example; §3.1 adds the notation table and practical reading of the reference width; Figures 1–4 are cited and explained; the supplement is merged into the PDF; and Appendix T reports the completed real-outcome analyses in detail.

Revision status. We have uploaded a revised PDF; major additions and corrections are marked in blue, and the former supplementary material is now included as appendices in the same PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.