Identifiability-Aware Uncertainty Audits for Sparse Leaderboards

Modern evaluation leaderboards often combine noisy results from partially overlapping views, yet uncertainty is usually assessed only after the evaluation table is fixed. We show that the cross-view covariances may not determine a unique shared scale for confidence intervals and ranking probabilities. We represent view overlap as a graph and prove that each bipartite connected component leaves one scale ambiguity, while a component with an odd cycle does not. We also derive an information bound under a Gaussian model for the estimated log cross-view moments, showing how weak graph conditioning limits the precision with which the scale can be recovered. The same graph structure also provides model-consistency checks and guides which new overlaps would be most informative to add. Building on our analysis, we propose anchored cross-view empirical Bayes (ACEB), a target-label-free uncertainty audit that combines graph diagnostics, fixed-anchor latent-score estimation, leave-view residual-noise estimation, and selective decisions that abstain when confidence is low. In sparse heterogeneous simulations, ACEB improves uncertainty diagnostics over several target-free baselines. After same-coverage recalibration, ACEB variants achieve the lowest pairwise Brier and interval scores among the six compared methods. On 13,077 MedHELM pairwise-gap records, coverage recalibration gives ACEB 22% fewer false signed declarations per record than random effects (0.147 versus 0.189). ACEB makes signed decisions on 78.6% rather than 98.3% of records, indicating that most of reduction comes from greater abstention. Together, these results show that accounting for overlap structure can inform both leaderboard uncertainty quantification and selective decision making.