acceptodds
Under review as a conference paper at ICLR 2027

Identifiability-Aware Uncertainty Audits for Sparse Leaderboards

Abstract

Modern evaluation leaderboards often combine noisy results from partially overlapping views, yet uncertainty is usually assessed only after the evaluation table is fixed. We show that the cross-view covariances may not determine a unique shared scale for confidence intervals and ranking probabilities. We represent view overlap as a graph and prove that each bipartite connected component leaves one scale ambiguity, while a component with an odd cycle does not. We also derive an information bound under a Gaussian model for the estimated log cross-view moments, showing how weak graph conditioning limits the precision with which the scale can be recovered. The same graph structure also provides model-consistency checks and guides which new overlaps would be most informative to add. Building on our analysis, we propose anchored cross-view empirical Bayes (ACEB), a target-label-free uncertainty audit that combines graph diagnostics, fixed-anchor latent-score estimation, leave-view residual-noise estimation, and selective decisions that abstain when confidence is low. In sparse heterogeneous simulations, ACEB improves uncertainty diagnostics over several target-free baselines. After same-coverage recalibration, ACEB variants achieve the lowest pairwise Brier and interval scores among the six compared methods. On 13,077 MedHELM pairwise-gap records, coverage recalibration gives ACEB 22% fewer false signed declarations per record than random effects (0.147 versus 0.189). ACEB makes signed decisions on 78.6% rather than 98.3% of records, indicating that most of reduction comes from greater abstention. Together, these results show that accounting for overlap structure can inform both leaderboard uncertainty quantification and selective decision making.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
uncertainty quantification, identifiability, empirical Bayes, latent-variable models, benchmark evaluation, selective decisions

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_L9oWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies uncertainty issue of evaluation leaderboards where only partially overlapping views are available. In such situation, the cross-view covariances may be underdetermined because of sparse evaluations. This paper interpret view overlap as a graph and prove that each bipartite connected component leaves one scale ambiguity. Based on analysis, the paper proposed anchored cross-view empirical Bayes, which by adding edges/ repairing the graph, the graph becomes identifiable. The edge parameter error is bounded by a constant inversely proportional to the minimum singular value of the anchored edge matrix. When the anchored edge matrix is ill-posed, the error could be amplified to infinity. Simulation results validate that the proposed method reduces overconfidence relative to target-free baselines.

Critical strengths

The paper provided a graph perspective of the sparse leaderboards problem with theoretical analysis using linear Gaussian models. The graph arguments appear correct within the specified moment model, validated by simulation results. It also honestly described the limitations of the work in the discussion part.

Critical weaknesses

The novelty of the paper is limited. Ma et al. already establish the same criterion that identification requires the absence of bipartite connected components, together with sample-complexity analysis involving the signless Laplacian. The present paper appropriately cites this connection. Theorem 3.2 is the standard Gaussian linear-model information bound under the assumed log-moment experiment. The cycle checks and single-edge eigenvalue derivative follow directly from the linear-algebra formulation. Similar ideas can be found in many related papers dealing with ill-posed problems such as Ridge regularization. The paper should identify the precise new technical obstacle or result.

Questions for authors
  1. What numerical output changes when the audit detects structural deficiency or poor conditioning?
  2. What is the principal new result beyond existing rank-one identification theory and standard factor-model estimation?
2 comments

We thank the reviewer for the constructive feedback. Below, we address each concern with additional clarifications.

Q1 What numerical output changes when the audit detects structural deficiency or poor conditioning?

Response. We clarify that graph warnings do not directly change ACEB's posterior estimates or decisions. Instead, the audit quantifies whether cross-view moments identify a shared scale and how reliably that scale can be estimated. We use these diagnostics to identify informative additional evaluations, whose benefits we validate through paired graph interventions.

  • Graph diagnostics quantify scale identification and stability.

    We report rank deficiency δ(G)\delta(G), the normalized smallest singular value, and flags for nonpositive cross-view moments (Table 2). Positive deficiency indicates unresolved scale directions, while a small positive singular value indicates sensitivity to moment-estimation error. The full observation model may provide additional information (Appendix C.1).

    For fixed observations, reference view, and fitting settings, graph warnings do not alter posterior means, variances, comparison probabilities, or decisions (Algorithm 1).

  • Graph diagnostics guide evaluations that improve estimation.

    Consider the three-view star in Figure 1, where view 0 overlaps with views 1 and 2, but the latter two do not overlap. The audit reports δ(G)=1\delta(G)=1 and a zero smallest singular value. Adding the missing overlap closes a triangle, giving δ(G)=0\delta(G)=0 and a positive smallest singular value.

    We also validate this guidance empirically. In paired synthetic interventions, adding overlaps that repair deficient designs improves loading recovery, latent-score RMSE, and pairwise Brier score on average (Section 4.1; Appendix D.10).

    We further test whether graph diagnostics improve prediction of ACEB's remaining calibration error beyond full-model fitting and leave-view noise correction. Across 840 synthetic fits, we use five-fold cross-validation to predict the absolute 90% coverage gap. We report the mean absolute prediction error (MAE) for ACEB's coverage gap, comparing a baseline model that accounts for simulation settings, observation availability, and reported standard errors with the same model with graph diagnostics added.

    DesignsBase MAE+Graph MAEReduction
    All (840)0.051830.048745.96%
    Connected (720)0.046740.044963.81%
    Full rank (420)0.042960.042850.26%

    We find that graph diagnostics reduce held-out prediction error by 5.96% overall, with improvements in all five evaluation folds. A paired bootstrap gives a 95% interval of [0.00223,0.00394][0.00223,0.00394] for the absolute MAE reduction.

    Thus, the audit provides actionable information about where additional evaluations are needed. Graph warnings do not change current decisions, but collecting new observations and refitting can improve subsequent inference.

Weakness & Q2
What is the principal new result beyond existing rank-one identification theory and standard factor-model estimation?

Response.
We thank the reviewer for highlighting the connection to prior work. We acknowledge that our graph-identification criterion builds on Ma et al. (2020), and would like to highlight the following major methodological differences from existing literature.

Our principal methodological contribution is a target-label-free empirical-Bayes uncertainty audit for sparse continuous leaderboards. Unlike Ma et al.'s recovery of worker skills from binary-label correlations, we model continuous scores with a Gaussian latent-score prior, view-specific sensitivities, reported sampling variances, and unknown additional view noise.

We address a distinct inference problem: recovering view scales alone does not provide the noise estimates needed for reliable posterior uncertainty. ACEB tackles this through staged fitting and leave-view residual-noise estimation, producing posterior intervals and comparison probabilities. We further combine these estimates with graph diagnostics that identify missing scale information and guide additional evaluations.

  • Continuous uncertainty requires more than recovering rank-one structure.
    Ma et al. estimate worker skills from correlations between binary labels and use them for classification. We instead model continuous scores using a Gaussian latent-score prior, view-specific sensitivities, reported sampling variances sij2s_{ij}^2, and unknown additional view variances τj2\tau_j^2 (Section 2).

    Even when cross-view moments Mjk=vajakM_{jk}=va_ja_k identify the view scales, these moments alone do not determine the additional view variances τj2\tau_j^2 needed for posterior uncertainty. Our empirical-Bayes model provides plug-in intervals and comparison probabilities, rather than only estimates of view scales.

  • Leave-view estimation addresses residual-noise uncertainty.
    Rather than jointly optimizing all noise parameters, we estimate additional view noise using predictions that exclude each observation's own view, correcting for both sampling variance and latent prediction uncertainty (Eq. (6)).

    Proposition B.1 establishes consistency under stated conditions. In synthetic experiments, this correction improves 90% coverage from 66.65% to 84.17% with identical fitted priors and loadings.

  • Graph diagnostics expose missing information that regularization cannot supply.
    Consider the three-view star in Figure 1, where view 0 is the reference (a0=1a_0=1), vv is the latent-score variance, and MjkM_{jk} denotes the cross-view covariance. The observed moments M01=va1M_{01}=va_1 and M02=va2M_{02}=va_2 cannot uniquely determine vv. Ridge regularization can select a numerical solution but cannot resolve this ambiguity from the observed moments. Our audit identifies the missing overlap (1,2)(1,2), which closes a triangle and determines v=M01M02/M12v=M_{01}M_{02}/M_{12} under the working model.

    We validate this guidance through paired graph interventions: adding overlaps that repair deficient designs improves loading recovery, latent-score RMSE, and pairwise Brier score on average (Section 4.1; Appendix D.10).

    We further evaluate the added value of graph diagnostics across 840 synthetic fits after leave-view noise correction. For example, in the above response, we find that adding graph diagnostics reduces MAE from 0.05183 to 0.04874 (5.96%).

Our theoretical results support the graph audit.
Theorem 3.1 characterizes the componentwise identified set, with one unresolved scale direction per bipartite component. Theorem 3.2 quantifies how weak conditioning limits scale-estimation precision under the Gaussian log-moment model. Their role is to provide the theoretical foundation for diagnosing scale ambiguity, assessing conditioning, and guiding informative overlap collection.

Reviewer_tjPBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies uncertainty in sparse leaderboards, where evaluation units are observed through partially overlapping views. The authors analyze how the overlap graph affects whether a shared measurement scale is identifiable, and they propose ACEB as an uncertainty-auditing method. On synthetic data, ACEB gives better uncertainty calibration in several settings. On MedHELM, it produces fewer false signed declarations per record than random effects, but also abstains substantially more often.

Critical strengths
  • The paper addresses an important problem in leaderboard evaluation. The graph analysis is clearly stated with explicit assumptions and proofs. I also found the distinction between identification from cross-view moments and identification under the full observation model useful.

  • The experimental section includes component ablations and an anchor-sensitivity analysis. The paper is also fairly candid about the limitations of the method, especially the wider intervals and higher abstention rate.

Critical weaknesses
  1. The main MedHELM result is difficult to interpret because the declaration rates are different. After recalibration, ACEB declares on 78.6% of records versus 98.3% for random effects, while their false-sign rates are nearly identical. The reported 22% reduction in false declarations may therefore mainly reflect greater abstention. I think comparison at matched declaration rates would be useful. The recalibration also uses target labels from other groups, so this setting is not fully target-free.

  2. The real-data experiments may not fully represent naturally sparse leaderboards. MedHELM creates views by splitting instances and selecting subsets, while SummEval uses anonymous annotation slots. Both are reasonable test beds, but it remains unclear whether the same identifiability problems arise and affect decisions in naturally sparse benchmark.

  3. The connection between the graph theory and the empirical gains is unclear. The theorem concerns off-diagonal moments, whereas ACEB fits the full observation model, and Appendix C.1 shows that diagonal information can resolve the ambiguity. he paper does not show whether the graph analysis predicts calibration problems or what it adds to leave-view noise estimation.

  4. I am concerned that the optimization may stop too early. ACEB runs for only eight iterations in the main simulation experiments and three on MedHELM. Joint EB also has a very large z-score standard deviation. Running the methods for more iterations and reporting whether they converge cloud help show whether these differences come from the method or from incomplete optimization.

Questions for authors

See weakness.

4 comments

We thank the reviewer for the constructive feedback. Below, we address each concern with additional experiments and clarifications.

W1 Does ACEB's MedHELM advantage mainly reflect greater abstention? How do methods compare at matched declaration rates?

Response.
We thank the reviewer for highlighting abstention as a confounder. We address this concern through held-out matched-declaration analyses on MedHELM and new controlled sparse simulations.

Although MedHELM's gain largely reflects greater abstention, our new sparse heterogeneous simulations demonstrate significantly lower false-sign rates for ACEB at identical declaration rates. We also demonstrate improved uncertainty estimation without abstention on the naturally sparse WMT20 dataset.

  • On MedHELM, matched declarations isolate the abstention effect.

    On the same 13,077 records, we use higher-budget ACEB and five whole-scenario folds, keeping all metrics and model pairs from each scenario together. Each method estimates its variance multiplier on the other four folds. Within each evaluation fold, both declare the same fraction of comparisons, ranked by their own confidence.

    We report ACEB minus random-effects false-sign rates among declarations (percentage points, pp; negative favors ACEB). We use simultaneous 95% confidence intervals to account for comparing all seven declaration rates together:

    DeclaredDifference (pp)Simultaneous 95% CI
    5%0.00[−0.28,0.28][-0.28,0.28]
    10%0.00[−3.34,3.34][-3.34,3.34]
    20%0.00[−8.38,8.38][-8.38,8.38]
    30%-0.08[−9.71,9.56][-9.71,9.56]
    50%-0.20[−5.53,5.14][-5.53,5.14]
    78.6%-0.26[−4.65,4.12][-4.65,4.12]
    90%0.00[−4.03,4.03][-4.03,4.03]

    We construct these intervals using 1,000 paired scenario/model-ID bootstrap reweightings, accounting for shared scenarios and models. All seven intervals include zero. Excluding zero targets or restricting to confidently signed targets gives the same conclusion.

    We therefore find no reliable matched-declaration advantage on MedHELM, where greater abstention explains much of the wrong-declaration reduction.

  • New sparse simulations show a gain at fixed declaration rates.

    We evaluated 400 datasets (340 new, 60 reused) and six methods, including sparse heterogeneous designs and homogeneous controls. Sparse datasets have 1,000 items, five views, only two observations per item, four overlap structures, and balanced or uneven view availability.

    After identical five-fold held-out recalibration, we force methods to declare exactly 5%, 10%, 20%, 30%, or 50% of comparisons. In the sparse heterogeneous panel, we compare ACEB with random effects:

    DeclaredACEB FSRRandom-effects FSRDifference (pp)
    5%0.076%1.124%-1.05
    10%0.127%1.581%-1.45
    20%0.321%2.231%-1.91
    30%0.638%2.721%-2.08
    50%1.826%3.855%-2.03

    At 20% declarations, we find that ACEB achieves a 1.91 pp lower false-sign rate (FSR) than random effects (95% CI: [−2.62,−1.19][-2.62,-1.19] pp).

    We construct simultaneous confidence intervals accounting for all five declaration rates using paired bootstrap resampling. All five intervals favor ACEB. We conclude that ACEB can select more accurate comparisons without an abstention-rate advantage in these sparse settings.

  • WMT20 validates persistent raters and no-abstention gains.

    Unlike MedHELM's instance blocks or SummEval's anonymous crowd slots, WMT20 has six persistent raters across documents, with three of the six rating each system–segment.

    We evaluate all 2,604 system-pair comparisons without abstention:

    MethodAll pairs: CoverageAll pairs: Int. scoreHigh disagreement (651): CoverageHigh disagreement (651): Int. score
    ACEB leave-view90.40%3.67491.55%4.347
    Random effects90.59%4.61696.31%6.654

    At nearly equal full-sample coverage, we achieve a 20.4% lower interval score than random effects. In the highest-disagreement quartile, the reduction is 34.7%.

    Under document-fold recalibration to approximately 90% coverage, scores remain 3.625 (ACEB), 3.704 (ASH-like), and 4.609 (random effects). This supports better uncertainty estimation with genuinely persistent views, independently of abstention.

Although MedHELM's gain largely reflects abstention, we demonstrate accuracy gains at identical declaration rates in sparse simulations and interval-quality gains without abstention on WMT20. Thus, abstention alone does not explain ACEB's benefits in the tested settings.

Manuscript revisions.
We will add MedHELM's paired matched-declaration results to Section 4.3 and Appendix E.1.2, report the new simulations and WMT20 comparisons, and clarify the whole-scenario folds and the role of abstention.

W2 Do the findings generalize from constructed or anonymous views to naturally sparse data with persistent raters? Do similar identification issues arise and affect inference?

Response.
We thank the reviewer for distinguishing natural sparsity from constructed missingness. We address this concern with a new experiment on WMT20 English-German MQM evaluations, where professional raters have persistent identities and ratings are naturally missing.

On WMT20, we demonstrate that ACEB achieves near-nominal coverage and favorable interval scores on identical comparisons without abstention, including when observed raters strongly disagree. We separately test structural identification failures through paired graph-repair experiments.

  • New real-world WMT20 data provide naturally sparse views.

    Unlike MedHELM's instance blocks or SummEval's anonymous annotation slots, WMT20 has six persistent professional raters, but each system–segment item has only three ratings, leaving half of the possible observations naturally missing.

    We construct 2,604 system-pair comparisons from seven translation systems and 124 documents. We hold out one rating per comparison, leaving two observed ratings and one held-out target. The held-out rater also appears in other documents, allowing us to estimate rater-specific effects.

    We fit ACEB and four baselines using identical observed ratings and evaluate all comparisons without abstention.

  • ACEB improves interval quality at near-nominal coverage.

    Across all 2,604 comparisons, ACEB leave-view achieves 90.40% coverage for nominal 90% intervals, versus 90.59% for random effects, with a 20.4% lower interval score (3.674 versus 4.616).

    We also examine the 651 comparisons with the greatest observed-rater disagreement, selected without held-out targets:

    MethodCoverageInt. scoreWidth
    Inverse variance85.25%4.9233.530
    ASH-like85.56%4.4823.029
    Random effects96.31%6.6546.360
    ACEB in-sample89.40%4.7263.152
    ACEB leave-view91.55%4.3473.418

    We find that ACEB achieves the lowest interval score, reducing it by 34.7% and interval width by 46.3% relative to random effects.

    After separate document-fold recalibration to approximately 90% coverage, ACEB also retains the lowest overall interval score (3.625 versus 3.704 for ASH-like and 4.609 for random effects).

  • We separately validate natural sparsity and structural identification.

    Despite naturally missing ratings, WMT20's overlap graph remains full rank. Thus, naturally incomplete evaluations can still identify a shared scale from cross-view moments under our model.

    Importantly, in this real-data analysis, we show that ACEB's uncertainty-estimation benefits do not require structural graph deficiency.

    We separately examine structural deficiencies through paired graph interventions (Section 4.1; Appendix D.10). We find that adding overlaps that repair deficient designs improves loading recovery, latent-score RMSE, and pairwise Brier score on average.

    From these experiments, we establish ACEB's uncertainty benefits under natural sparsity and the practical value of repairing insufficient overlap structure.

Manuscript revisions.
We will add the WMT20 experiment to Section 4, with construction details and additional results in Appendix E.

W3 The graph theory concerns off-diagonal moments, while ACEB uses the full observation model. Since graph warnings do not directly affect decisions, do they predict calibration problems and contribute beyond leave-view noise estimation?

Response.
We thank the reviewer for highlighting the distinction between moment identification and full-model inference. We address this concern by testing whether graph diagnostics remain informative after ACEB's full-model fitting and leave-view noise correction.

We find that adding graph diagnostics reduces held-out prediction error for ACEB's remaining coverage gap by 5.96%, beyond simulation settings and observation-availability controls. We also show how the graph audit identifies informative new overlaps, complementing leave-view noise estimation.

  • Clarification of the scope of graph identification.

    Under Eq. (8), cross-view moments satisfy Mjk=vajakM_{jk}=va_ja_k, where vv is latent-score variance and aja_j denotes view sensitivity. These moments provide scale information independently of additional view-noise variances.

    As Appendix C.1 shows, diagonal information in the full observation model can resolve ambiguities left by cross-view moments. Thus, graph deficiency signals insufficient identification from cross-view moments, not necessarily from the full model.

  • New evidence beyond leave-view estimation.

    We refitted all 840 synthetic cells with larger optimization budgets and measured each fit's absolute raw 90% coverage gap, ∣C^−0.90∣|\widehat C-0.90|.

    Using nested five-fold cross-validation grouped by simulation seed, we compared predictive models that control for simulation settings, observation allocation, per-view counts, and sampling errors. We then added graph rank and conditioning diagnostics:

    DesignsOutcomeBase MAE+Graph MAEReduction
    All (840)Coverage gap0.051830.048745.96%
    Connected (720)Coverage gap0.046740.044963.81%
    Full rank (420)Coverage gap0.042960.042850.26%
    Full rank (420)Log-loading RMSE0.093490.089304.48%

    We observe improvements in all five held-out folds for the overall and connected-design analyses. Using 2,000 paired block bootstrap replicates, we obtain 95% intervals of [0.002227,0.003940][0.002227,0.003940] and [0.000971,0.002613][0.000971,0.002613] for their absolute MAE reductions, respectively.

    Within full-rank designs, we find that conditioning adds little to coverage-error prediction (0.26%) but improves prediction of log-loading RMSE by 4.48%, consistent with Theorem 3.2's focus on scale-estimation precision.

    From these results, we demonstrate that graph diagnostics provide additional predictive information about ACEB's remaining calibration error, beyond observation-availability controls and leave-view noise estimation.

  • Graph diagnostics identify actionable measurement gaps.

    Leave-view estimation estimates additional view noise (Eq. (6)), while the graph audit identifies scale information missing from cross-view moments and guides which additional evaluations to collect.

    For example, consider the three-view star in Figure 1, where view 0 is the reference (a0=1a_0=1), vv denotes the latent-score variance, and MjkM_{jk} is the cross-view covariance. With only overlaps (0,1)(0,1) and (0,2)(0,2), the available moments M01=va1M_{01}=va_1 and M02=va2M_{02}=va_2 cannot uniquely determine vv.

    Regularization can select a numerical solution but cannot recover this missing moment information. Our audit identifies overlap (1,2)(1,2) as a repair, closing a triangle and determining v=M01M02/M12v=M_{01}M_{02}/M_{12} under the working model.

    We validate this guidance empirically through paired graph interventions. We find that adding overlaps that repair deficient designs improves loading recovery, latent-score RMSE, and pairwise Brier score on average (Section 4.1; Appendix D.10).

    We clarify that graph warnings do not directly alter posterior estimates or decisions (Algorithm 1). Instead, they complement leave-view estimation by identifying scale limitations, predicting remaining calibration difficulty, and guiding more informative evaluation designs.

Manuscript revisions.
We will add the predictive analysis to Section 4.1 and Appendix D, and clarify the distinct roles of graph diagnostics and posterior decisions.

W4 Do the synthetic and MedHELM findings persist with larger optimization budgets? What caused joint EB's extreme z-score dispersion?

Response.
We thank the reviewer for raising this optimization concern. Through additional higher-budget refitting and convergence diagnostics, we identify the cause of joint EB's original numerical failure and confirm that ACEB's key empirical findings persist across different optimization budgets.

  • Diagnosis of joint EB's original numerical failure.

Under the original 8-iteration cap, 839/840 joint EB fits terminated unsuccessfully. Its near-zero noise initialization (τj=10−4\tau_j=10^{-4}) scales the log-noise gradient by 2τj2=2×10−82\tau_j^2=2\times10^{-8}. With stabilized high-budget fitting (up to 100 iterations), joint EB's synthetic z-score SD decreased from 493.951 to 1.273. ACEB instead estimates noise through leave-view residual moments (Eq.~6), avoiding this gradient bottleneck.

We compared ACEB and stabilized joint EB under held-out same-coverage recalibration, varying only the iteration cap while keeping other fitting settings fixed. The following table reports mean RMSE and interval scores for synthetic simulations and MedHELM at approximately 90% coverage (caps 50 and 100 yield identical results):

  • ACEB achieves lower RMSE in synthetic dataset at convergence.

    CapACEB coverageJoint EB coverageACEB RMSEJoint EB RMSEACEB int. scoreJoint EB int. score
    1089.99%90.04%0.6820.5044.7495.002
    5090.00%90.02%0.3120.3151.9781.813
    10090.00%90.02%0.3120.3151.9781.813

    At matched coverage, we find that ACEB achieves lower RMSE to joint EB at convergence (cap 100: 0.312 vs. 0.315). Although joint EB achieves a lower interval score at convergence, ACEB achieves a lower interval score under the smaller optimization budget.

  • ACEB achieves lower RMSE on MedHELM across optimization budgets.

    We repeat the cap-controlled comparison across six MedHELM designs, reporting mean RMSE and held-out recalibrated interval scores:

    CapACEB coverageJoint EB coverageACEB RMSEJoint EB RMSEACEB int. scoreJoint EB int. score
    1090.07%90.57%0.9591.6311.4792.774
    5089.98%90.78%1.4101.5472.9472.538
    10089.98%90.78%1.4101.5472.9472.538

    At approximately 90% coverage, we find that ACEB achieves 8.86% lower RMSE than joint EB at convergence, although joint EB achieves a lower interval score.

Manuscript revisions.
We will add convergence diagnostics and higher-budget comparisons to the appendix and update the corresponding experimental discussion.

Reviewer_do6WOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Under a positive one-factor model, the paper characterizes when sparse cross-view covariances identify view scales and develops conditioning diagnostics, model checks, and overlap-repair criteria. ACEB reports these diagnostics alongside fixed-anchor empirical-Bayes estimates, leave-view residual-noise estimates, and selective decisions that abstain when neither direction has sufficient posterior confidence. Synthetic experiments examine calibration and graph repair. On MedHELM, fewer false declarations mainly accompany greater abstention. On SummEval, ACEB improves raw coverage over some baselines but remains undercovered before recalibration.

Critical strengths
  1. Theorem 3.1 and Propositions C.5–C.6 connect scale ambiguity, model compatibility, and informative new overlaps. This helps benchmark designers distinguish collecting more measurements from collecting the overlaps needed for identification.

  2. Ablations, the dense homogeneous negative control, and paired repairs probe the method's behavior. In the separate same-coverage experiment using a 100-iteration cap, ACEB in-sample achieves the lowest mean Brier score and ACEB leave-view the lowest mean interval score among the six included methods.

Critical weaknesses
  1. ACEB uses iteration caps of eight in the main synthetic benchmark and three on MedHELM, without reported convergence diagnostics (Table 7). Joint EB, which also estimates loadings and residual noise, has extreme z-score dispersion of unresolved cause (493.951493.951\pm444.185444.185; Table 3). It is excluded from the separately fitted, 100-iteration same-coverage comparison (D.5). Thus, the evidence does not yet separate the benefit of the staged procedure from numerical fitting choices.

  2. After recalibration, ACEB reduces false declarations per record from 0.189 to 0.147 versus random effects, while declaring on 78.6% rather than 98.3% of records. Conditional false-sign rates are similar: 0.187 versus 0.192. ACEB also has a worse interval score, 0.937 versus 0.792 (Table 5). The raw-audit curves in Figure 4C also favor IV and ASH-like at comparable declaration rates, motivating this comparison after recalibration to establish an advantage over raising baseline thresholds. Moreover, random instance blocks in MedHELM and anonymous crowd slots in SummEval provide limited validation of persistent heterogeneous views (E.1.1, E.2).

Questions for authors
  1. Do the main synthetic and MedHELM findings persist with larger optimization budgets and alternative initializations? Please report termination diagnostics for ACEB and joint EB, and include a stable joint-EB fit in the same-coverage comparison. This would clarify whether the reported advantage depends on numerical fitting choices.

  2. Under the same held-out recalibration, how do methods compare at equal declaration rates? Please report paired uncertainty estimates for the ACEB–random-effects differences, accounting for shared scenarios and models. Please also clarify whether all metrics from each scenario remain in one calibration fold. A consistent improvement would strengthen the practical contribution; comparable performance would support an interpretation centered on greater abstention.

4 comments

We thank the reviewer for the constructive feedback. Below, we address each concern with additional experiments and clarifications.

Q1 Do the synthetic and MedHELM findings persist with larger optimization budgets and alternative initializations? What caused joint EB's extreme z-score dispersion, and how does stabilized joint EB perform in the same-coverage comparison?

Response.
We thank the reviewer for raising the optimization concern. Through additional refitting and convergence diagnostics, we identify the cause of joint EB's original numerical failure and demonstrate ACEB's advantages in raw calibration, computational efficiency, and termination stability across the tested initializations relative to stabilized joint EB.

  • Diagnosis of joint EB's original numerical failure.

Under the original 8-iteration cap, 839/840 joint EB fits terminated unsuccessfully. Its near-zero noise initialization (τj=10−4\tau_j=10^{-4}) scales the log-noise gradient by 2τj2=2×10−82\tau_j^2=2\times10^{-8}. Increasing the budget to 100 iterations and using positive-noise initialization reduced its synthetic z-score SD from 493.951 to 1.273. ACEB instead estimates noise through leave-view residual moments (Eq. 6), avoiding this gradient bottleneck.

We refitted all 840 synthetic cells with budgets up to 100 iterations, and tested 5 initializations on 60 synthetic datasets and all 6 MedHELM designs. Both methods yielded stable selected synthetic fits. Joint EB's largest per-fit 99th-percentile changes between final budgets were 0.74% in posterior SD and 0.0091 posterior-SD units in posterior means. ACEB's posteriors and decisions were unchanged.

  • ACEB offers better raw calibration at lower cost.

    On the same 60 synthetic datasets, we compare ACEB with stabilized joint EB. Fit times include residual calibration and posterior computation.

    MetricACEBJoint EB
    Raw 90% coverage91.75%78.94%
    Mean absolute coverage error (pp)9.1111.29
    Mean absolute z-score SD error0.4290.464
    Median synthetic fit time (s)0.0160.036
    Median MedHELM fit time (s)0.2431.019

    Across three timing runs per tested setting, we find that ACEB was 2.3× faster on synthetic data and 4.2× faster on MedHELM.

  • ACEB achieves lower RMSE in synthetic dataset at convergence.

    We compared ACEB and stabilized joint EB under held-out same-coverage recalibration, varying only the iteration cap while keeping other fitting settings fixed. We report mean RMSE and recalibrated interval scores at matched coverage:

    CapACEB coverageJoint EB coverageACEB RMSEJoint EB RMSEACEB int. scoreJoint EB int. score
    1089.99%90.04%0.6820.5044.7495.002
    5090.00%90.02%0.3120.3151.9781.813
    10090.00%90.02%0.3120.3151.9781.813

    At matched coverage, we find that ACEB achieves comparable RMSE to joint EB at convergence (cap 100: 0.312 vs. 0.315). Although joint EB achieves a lower interval score at convergence, ACEB achieves a lower interval score under the smaller optimization budget.

  • ACEB achieves lower RMSE on MedHELM across all optimization budgets.

    We repeat the cap-controlled comparison across 6 MedHELM designs, reporting mean RMSE and held-out recalibrated interval scores:

    CapACEB coverageJoint EB coverageACEB RMSEJoint EB RMSEACEB int. scoreJoint EB int. score
    1090.07%90.57%0.9591.6311.4792.774
    5089.98%90.78%1.4101.5472.9472.538
    10089.98%90.78%1.4101.5472.9472.538

    At approximately 90% coverage, we find that ACEB achieves 8.86% lower RMSE than joint EB at convergence, although joint EB achieves a lower interval score.

Manuscript revisions.
We will update synthetic and MedHELM results in the main text and add convergence, timing, and stable joint EB comparisons to the appendix.

Q2 Does ACEB's MedHELM advantage mainly reflect greater abstention? How do methods compare at matched declaration rates?

Response.
We thank the reviewer for highlighting abstention as a confounder. We address this concern through held-out matched-declaration analyses on MedHELM and new controlled sparse simulations.

Although MedHELM's gain largely reflects greater abstention, our new sparse heterogeneous simulations demonstrate significantly lower false-sign rates for ACEB at identical declaration rates. We also demonstrate improved uncertainty estimation without abstention on the naturally sparse WMT20 dataset.

  • On MedHELM, matched declarations isolate the abstention effect.

    On the same 13,077 records, we use higher-budget ACEB and five whole-scenario folds, keeping all metrics and model pairs from each scenario together. Each method estimates its variance multiplier on the other four folds. Within each evaluation fold, both declare the same fraction of comparisons, ranked by their own confidence.

    We report ACEB minus random-effects false-sign rates among declarations (percentage points, pp; negative favors ACEB). We use simultaneous 95% confidence intervals to account for comparing all seven declaration rates together:

    DeclaredDifference (pp)Simultaneous 95% CI
    5%0.00[−0.28,0.28][-0.28,0.28]
    10%0.00[−3.34,3.34][-3.34,3.34]
    20%0.00[−8.38,8.38][-8.38,8.38]
    30%-0.08[−9.71,9.56][-9.71,9.56]
    50%-0.20[−5.53,5.14][-5.53,5.14]
    78.6%-0.26[−4.65,4.12][-4.65,4.12]
    90%0.00[−4.03,4.03][-4.03,4.03]

    We construct these intervals using 1,000 paired scenario/model-ID bootstrap reweightings, accounting for shared scenarios and models. All seven intervals include zero. Excluding zero targets or restricting to confidently signed targets gives the same conclusion.

    We therefore find no reliable matched-declaration advantage on MedHELM, where greater abstention explains much of the wrong-declaration reduction.

  • New sparse simulations show a gain at fixed declaration rates.

    We evaluated 400 datasets (340 new, 60 reused) and six methods, including sparse heterogeneous designs and homogeneous controls. Sparse datasets have 1,000 items, five views, only two observations per item, four overlap structures, and balanced or uneven view availability.

    After identical five-fold held-out recalibration, we force methods to declare exactly 5%, 10%, 20%, 30%, or 50% of comparisons. In the sparse heterogeneous panel, we compare ACEB with random effects:

    DeclaredACEB FSRRandom-effects FSRDifference (pp)
    5%0.076%1.124%-1.05
    10%0.127%1.581%-1.45
    20%0.321%2.231%-1.91
    30%0.638%2.721%-2.08
    50%1.826%3.855%-2.03

    At 20% declarations, we find that ACEB achieves a 1.91 pp lower false-sign rate (FSR) than random effects (95% CI: [−2.62,−1.19][-2.62,-1.19] pp).

    We construct simultaneous confidence intervals accounting for all five declaration rates using paired bootstrap resampling. All five intervals favor ACEB. We conclude that ACEB can select more accurate comparisons without an abstention-rate advantage in these sparse settings.

  • WMT20 validates persistent raters and no-abstention gains.

    Unlike MedHELM's instance blocks or SummEval's anonymous crowd slots, WMT20 has six persistent raters across documents, with three of the six rating each system–segment.

    We evaluate all 2,604 system-pair comparisons without abstention:

    MethodAll pairs: CoverageAll pairs: Int. scoreHigh disagreement (651): CoverageHigh disagreement (651): Int. score
    ACEB leave-view90.40%3.67491.55%4.347
    Random effects90.59%4.61696.31%6.654

    At nearly equal full-sample coverage, we achieve a 20.4% lower interval score than random effects. In the highest-disagreement quartile, the reduction is 34.7%.

    Under document-fold recalibration to approximately 90% coverage, scores remain 3.625 (ACEB), 3.704 (ASH-like), and 4.609 (random effects). This supports better uncertainty estimation with genuinely persistent views, independently of abstention.

Although MedHELM's gain largely reflects abstention, we demonstrate accuracy gains at identical declaration rates in sparse simulations and interval-quality gains without abstention on WMT20. Thus, abstention alone does not explain ACEB's benefits in the tested settings.

Manuscript revisions.
We will add MedHELM's paired matched-declaration results to Section 4.3 and Appendix E.1.2, report the new simulations and WMT20 comparisons, and clarify the whole-scenario folds and the role of abstention.

Thank you for the additional experiments. The convergence checks address my optimization concern. On MedHELM, the paired analysis supports the manuscript’s interpretation that greater abstention explains most of the reduction in false declarations. At matched declaration rates, ACEB showed no statistically reliable improvement in false-sign rates over random effects. The rebuttal also shows that, with a 100-iteration cap, ACEB had higher recalibrated interval scores than stabilized joint EB at comparable coverage on both synthetic data (1.978 vs. 1.813) and MedHELM (2.947 vs. 2.538). I therefore maintain my original rating.

We thank the reviewer for their thoughtful follow-up. We would like to further clarify our previous response to address your concern.

First, the absence of a statistically reliable matched-declaration improvement is specific to MedHELM. MedHELM constructs views by splitting instances, which may not fully capture the heterogeneity of naturally sparse leaderboards.

To clarify, we additionally evaluated real-world dataset WMT20, which contains naturally missing ratings from persistent raters. Without abstention, ACEB achieves a 20.4% lower interval score than random effects (3.674 vs. 4.616) at nearly identical coverage. Furthermore, our sparse heterogeneous simulations demonstrate significantly lower false-sign rates at identical declaration rates. At 50% declarations, ACEB achieves 1.826% versus 3.855% for random effects (difference: −2.03 percentage points). Paired-bootstrap simultaneous 95% confidence intervals favor ACEB at all five tested declaration rates (see the table in our previous response for Q2 in second bullet point for detailed results). These findings demonstrate that greater abstention does not fully explain ACEB's benefits across the tested settings.

Second, regarding the higher-budget comparison, we acknowledge that stabilized joint EB achieves lower recalibrated interval scores at convergence. However, in synthetic experiment, even without recalibration, ACEB achieves substantially better raw coverage (91.75% vs. 78.94%) and lower mean absolute coverage error (9.11 vs. 11.29 percentage points). Under same-coverage recalibration, ACEB also achieves lower RMSE on both synthetic data (0.312 vs. 0.315) and MedHELM (1.410 vs. 1.547), with an 8.86% reduction on MedHELM.

We thank the reviewer's time and consideration, and hope our explanations further clarify the concerns raised in the response.

Comments on the paper

1 comment

Thank you again for your time and valuable feedback on our paper. As the rebuttal discussion period is approaching its end, we would greatly appreciate any further comments or feedback on our responses, particularly if there are any remaining concerns we could clarify.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.