- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies uncertainty issue of evaluation leaderboards where only partially overlapping views are available. In such situation, the cross-view covariances may be underdetermined because of sparse evaluations. This paper interpret view overlap as a graph and prove that each bipartite connected component leaves one scale ambiguity. Based on analysis, the paper proposed anchored cross-view empirical Bayes, which by adding edges/ repairing the graph, the graph becomes identifiable. The edge parameter error is bounded by a constant inversely proportional to the minimum singular value of the anchored edge matrix. When the anchored edge matrix is ill-posed, the error could be amplified to infinity. Simulation results validate that the proposed method reduces overconfidence relative to target-free baselines.
Critical strengths
The paper provided a graph perspective of the sparse leaderboards problem with theoretical analysis using linear Gaussian models. The graph arguments appear correct within the specified moment model, validated by simulation results. It also honestly described the limitations of the work in the discussion part.
Critical weaknesses
The novelty of the paper is limited. Ma et al. already establish the same criterion that identification requires the absence of bipartite connected components, together with sample-complexity analysis involving the signless Laplacian. The present paper appropriately cites this connection. Theorem 3.2 is the standard Gaussian linear-model information bound under the assumed log-moment experiment. The cycle checks and single-edge eigenvalue derivative follow directly from the linear-algebra formulation. Similar ideas can be found in many related papers dealing with ill-posed problems such as Ridge regularization. The paper should identify the precise new technical obstacle or result.
Questions for authors
- What numerical output changes when the audit detects structural deficiency or poor conditioning?
- What is the principal new result beyond existing rank-one identification theory and standard factor-model estimation?
2 comments
Weakness & Q2
What is the principal new result beyond existing rank-one identification theory and standard factor-model estimation?
Response.
We thank the reviewer for highlighting the connection to prior work. We acknowledge that our graph-identification criterion builds on Ma et al. (2020), and would like to highlight the following major methodological differences from existing literature.
Our principal methodological contribution is a target-label-free empirical-Bayes uncertainty audit for sparse continuous leaderboards. Unlike Ma et al.'s recovery of worker skills from binary-label correlations, we model continuous scores with a Gaussian latent-score prior, view-specific sensitivities, reported sampling variances, and unknown additional view noise.
We address a distinct inference problem: recovering view scales alone does not provide the noise estimates needed for reliable posterior uncertainty. ACEB tackles this through staged fitting and leave-view residual-noise estimation, producing posterior intervals and comparison probabilities. We further combine these estimates with graph diagnostics that identify missing scale information and guide additional evaluations.
-
Continuous uncertainty requires more than recovering rank-one structure.
Ma et al. estimate worker skills from correlations between binary labels and use them for classification. We instead model continuous scores using a Gaussian latent-score prior, view-specific sensitivities, reported sampling variances , and unknown additional view variances (Section 2).Even when cross-view moments identify the view scales, these moments alone do not determine the additional view variances needed for posterior uncertainty. Our empirical-Bayes model provides plug-in intervals and comparison probabilities, rather than only estimates of view scales.
-
Leave-view estimation addresses residual-noise uncertainty.
Rather than jointly optimizing all noise parameters, we estimate additional view noise using predictions that exclude each observation's own view, correcting for both sampling variance and latent prediction uncertainty (Eq. (6)).Proposition B.1 establishes consistency under stated conditions. In synthetic experiments, this correction improves 90% coverage from 66.65% to 84.17% with identical fitted priors and loadings.
-
Graph diagnostics expose missing information that regularization cannot supply.
Consider the three-view star in Figure 1, where view 0 is the reference (), is the latent-score variance, and denotes the cross-view covariance. The observed moments and cannot uniquely determine . Ridge regularization can select a numerical solution but cannot resolve this ambiguity from the observed moments. Our audit identifies the missing overlap , which closes a triangle and determines under the working model.We validate this guidance through paired graph interventions: adding overlaps that repair deficient designs improves loading recovery, latent-score RMSE, and pairwise Brier score on average (Section 4.1; Appendix D.10).
We further evaluate the added value of graph diagnostics across 840 synthetic fits after leave-view noise correction. For example, in the above response, we find that adding graph diagnostics reduces MAE from 0.05183 to 0.04874 (5.96%).
Our theoretical results support the graph audit.
Theorem 3.1 characterizes the componentwise identified set, with one unresolved scale direction per bipartite component. Theorem 3.2 quantifies how weak conditioning limits scale-estimation precision under the Gaussian log-moment model. Their role is to provide the theoretical foundation for diagnosing scale ambiguity, assessing conditioning, and guiding informative overlap collection.
We thank the reviewer for the constructive feedback. Below, we address each concern with additional clarifications.
Q1 What numerical output changes when the audit detects structural deficiency or poor conditioning?
Response. We clarify that graph warnings do not directly change ACEB's posterior estimates or decisions. Instead, the audit quantifies whether cross-view moments identify a shared scale and how reliably that scale can be estimated. We use these diagnostics to identify informative additional evaluations, whose benefits we validate through paired graph interventions.
Graph diagnostics quantify scale identification and stability.
We report rank deficiency δ(G), the normalized smallest singular value, and flags for nonpositive cross-view moments (Table 2). Positive deficiency indicates unresolved scale directions, while a small positive singular value indicates sensitivity to moment-estimation error. The full observation model may provide additional information (Appendix C.1).
For fixed observations, reference view, and fitting settings, graph warnings do not alter posterior means, variances, comparison probabilities, or decisions (Algorithm 1).
Graph diagnostics guide evaluations that improve estimation.
Consider the three-view star in Figure 1, where view 0 overlaps with views 1 and 2, but the latter two do not overlap. The audit reports δ(G)=1 and a zero smallest singular value. Adding the missing overlap closes a triangle, giving δ(G)=0 and a positive smallest singular value.
We also validate this guidance empirically. In paired synthetic interventions, adding overlaps that repair deficient designs improves loading recovery, latent-score RMSE, and pairwise Brier score on average (Section 4.1; Appendix D.10).
We further test whether graph diagnostics improve prediction of ACEB's remaining calibration error beyond full-model fitting and leave-view noise correction. Across 840 synthetic fits, we use five-fold cross-validation to predict the absolute 90% coverage gap. We report the mean absolute prediction error (MAE) for ACEB's coverage gap, comparing a baseline model that accounts for simulation settings, observation availability, and reported standard errors with the same model with graph diagnostics added.
We find that graph diagnostics reduce held-out prediction error by 5.96% overall, with improvements in all five evaluation folds. A paired bootstrap gives a 95% interval of [0.00223,0.00394] for the absolute MAE reduction.
Thus, the audit provides actionable information about where additional evaluations are needed. Graph warnings do not change current decisions, but collecting new observations and refitting can improve subsequent inference.