acceptodds
Under review as a conference paper at ICLR 2027

Auditing Common-Threshold Extrapolation for Rare Events: Identifiability, Prediction, and Failure Modes

Abstract

When can common-threshold labels improve estimation of a rare-event probability? We develop a target-specific audit that distinguishes identifiability, statistical efficiency, and learning performance. For a locally identifiable target, the endpoint-to-common efficient variance ratio is , where is the target log-probability gradient and is common-label Fisher information. A target direction outside the Fisher range rules out regular root- estimation. Spectral diagnostics and a perturbation bound assess the stability of estimated information. In a two-rate benchmark, common-only extrapolation improves deep-tail accuracy, misspecification reverses the gain, and an added threshold restores identifiability. A prospective neural study evaluates five targets at three model sizes with approximately matched parameter counts. A pointwise Fisher score predicts the mean direction of the error difference in all 12 cells outside its indifference band, with rank correlation 0.917 across all 15 cells. At the deepest target, endpoint-to-common error ratios are 1.67–1.74. An independent 20,000-observation common-label pilot identifies harm at three targets and abstains at two, falling short of its pre-specified decision-coverage criterion. Temporal tests show mixed effects on USGS earthquakes and 3.5–16.8% lower held-out log loss in four gas-turbine target–year cells. The resulting approach connects label design to testable learning predictions through target identifiability and information margins.

Primary area:
learning theory
Keywords:
rare-event estimation, Fisher information, identifiability, common-threshold labels, statistical efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8DR3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks when common-threshold labels can estimate a rare-event probability more accurately than observing the endpoint directly, at the same sample size. The authors propose a target-specific audit using Fisher-range identifiability, an endpoint-to-common asymptotic variance ratio, and diagnostics for estimated information. A two-rate benchmark shows deep-tail gains as well as failures caused by missing information and tail misspecification. In a prospective neural experiment, the Fisher score correctly predicts the direction of the mean error difference for four targets across three model sizes, while an independent pilot abstains on the two deepest targets. Results are mixed on USGS earthquake data, while common-only extrapolation achieves lower test log loss on gas-turbine data.

Critical strengths
  1. The paper separates whether common-threshold labels identify a rare target from whether they offer an efficiency advantage over endpoint observations. The Fisher-range condition and endpoint-relative variance ratio provide a concrete framework for this distinction, while the help/harm/abstain rule accounts for uncertainty in estimated information.
  2. Removing a threshold makes the target unidentifiable, while tail curvature introduces enough bias to reverse the common-only gain. Adding another threshold restores identifiability, but does not necessarily give an efficiency advantage.
  3. The authors provide a separate replication, bootstrap checks, and detailed protocols. They also report the failed near-crossover replication cell and the pilot's failure to meet its decision-coverage criterion.
Critical weaknesses
  1. The neural experiment is closely matched to the data-generating model. The DGP, QaQ_a, and common-only network all use the same two-hazard structure, with only one context variable. The 12/12 directional matches also come from just four target-level predictions evaluated at three network widths, rather than 12 independent tests. Moreover, only the large-model cell at a=8 has a positive bootstrap lower bound for the paired error difference. It remains unclear whether QaQ_a predicts gains with a different model or survival head.
  2. Much of the theory builds on established results. Theorem 1 draws on local asymptotic efficiency and c-optimal design, while Propositions 1–2 use standard spectral and matrix perturbation arguments. I see the main contribution in bringing these tools together into a target-specific audit, rather than in the individual theoretical results.
  3. The pilot does not resolve the strongest positive case. At a=8, where the neural experiment shows its largest common-only gain, the pilot estimates Q8=0.99Q_8=0.99 with an interval of [0.59,1.76][0.59,1.76], compared with the oracle value of 1.43, and abstains. It also fails the pre-specified decision-coverage criterion. Abstention is reasonable here, but it is unclear how much pilot data would be needed to make a reliable positive decision.
  4. The real-data experiments provide only limited validation of the full audit procedure. On USGS data, common-only estimation performs better in only 6, 6, and 7 of 16 yearly folds across the three targets. The gas-turbine results are more positive, but the experiment was designed after data inspection and covers only two test years. Although the fitted training-set efficiency ratios exceed one, the study does not report uncertainty bounds or pre-specified help/harm/abstain decisions. Temporal shifts also make it difficult to separate efficiency gains from changes in prediction bias.
Questions for authors

Q1. Could the authors repeat the Section 4.3 experiment with a different common-only survival head that does not match the DGP's two-hazard structure, while still allowing extrapolation from common labels? Keeping the data, targets, and model capacity comparable would make it easier to see whether QaQ_a remains predictive under model mismatch.

Q2. Across repeated independent pilots, how often would the lower confidence bound for Q8Q_8 exceed 1.05? A sample-size sweep reporting this positive-decision rate and interval coverage would help assess how much data the audit needs.

Q3. Could the authors check Eq. (3) against Table 2? The equation takes the square root of a ratio of averaged variances, whereas the primary metric averages absolute log errors across contexts. Under a pointwise Gaussian approximation, averaging pointwise standard deviations seems more directly related to this metric. Using the stated DGP, I obtain Q8≈1.48Q_8\approx1.48 from Eq. (3), but approximately 1.43 when averaging pointwise standard deviations, matching Table 2. Which aggregation was implemented, and does this choice affect the pilot decisions?

Q4. In the gas-turbine experiment, the training endpoint and common-only estimates at target 110 are 0.00782 and 0.00464. What are the observed exceedance frequencies in 2014 and 2015? Could the authors also report the audit's help/harm/abstain decisions, with uncertainty, using only the 2011–2013 common-label data? This may help separate an efficiency gain from improved calibration under temporal shift.

3 comments

Thank you for the specific suggestions on alternative survival heads, independent pilots, aggregation, and temporal event frequencies. We added the requested checks and report their statistical interpretation below.

Q1. Alternative head and matched controls

We ran a separate 24-seed, five-method study using 20,000 training and 5,000 validation observations per seed, the same targets, and three approximately parameter-matched capacities. E-direct uses endpoint labels; C-structured uses common labels with a two-hazard head; C-Weibull uses the same common-label likelihood with a Weibull head; E-structured uses the two-hazard architecture with endpoint supervision; and F-structured uses that architecture with joint common-plus-endpoint supervision.

The original common-only mean directions recur for all four decisive targets at every capacity. At a=8a=8, E-direct/C-structured mean-error ratios are 2.05, 2.21, and 2.19. Holding architecture fixed, F-structured improves over E-structured under the simultaneous comparison rule at a=8a=8 for all sizes and at a=7a=7 for the large model. Its target-8 ratios are 3.05-3.44. This quantifies the value of additional common-label information alongside endpoint labels.

The Weibull control distinguishes common-label fit from target extrapolation. Its population solution fits both common survivals exactly but has target-8 working-model score 2.846 and mean absolute log bias 0.982. Adjusted comparisons favor E-direct at targets 4-7; those at target 8 are inconclusive. Section 4.3 and Appendix D give all 75 method-capacity-target cells, 45 paired simultaneous comparisons, and the alternative-head specification. Original and follow-up seed cohorts are kept separate.

Q2. Independent pilots and sample requirements

For each of five sample sizes, we generated 200 independent pilots and used 2,000 row-bootstrap refits per pilot for simultaneous 95% intervals across five targets. Help requires the lower endpoint to exceed 1.05. At target 8:

Pilot nHelp rateExact 95% MC intervalFive-target coverage
20,00017.0%[12.1%, 22.9%]97.5%
50,00047.5%[40.4%, 54.7%]98.0%
100,00081.0%[74.9%, 86.2%]98.5%
200,00099.5%[97.2%, 100.0%]97.5%
500,000100.0%[98.2%, 100.0%]99.5%

Thus 199/200 pilots resolve model-based Help at the deepest target with 200,000 observations. Figure 2 and Appendix B report all targetwise decisions, joint coverage, the original pilot outcome, and the 5,000-refit sensitivity. Pilot data are additional to the neural training sample.

Q3. Equation (3) and same-draw comparison

Equation (3), its implementation, and Table 2A use the ratio of averaged pointwise standard deviations:

QSD=⟨Ve(x)⟩x⟨Vc(x)⟩x.Q_{\mathrm{SD}}=\frac{\langle\sqrt{V_e(x)}\rangle_x}{\langle\sqrt{V_c(x)}\rangle_x}.

For regular pointwise estimators under the stated Gaussian and absolute-moment conditions, expected absolute log error is proportional to SD, motivating this aggregation across contexts. At a=8a=8, the oracle value is 1.434293 (Table 2A: 1.43); the distinct RMS ratio QRMS=⟨Ve⟩x/⟨Vc⟩xQ_{\mathrm{RMS}}=\sqrt{\langle V_e\rangle_x/\langle V_c\rangle_x} is 1.476253.

We replayed both on the same 1,000 original bootstrap-fitted coefficient vectors. At target 8, SD gives 0.990560 [0.588590, 1.762004], while RMS gives 1.004206 [0.589879, 1.877100]. All five original audit decisions remain unchanged. Appendix D, Table 12 reports every target. Shared-network prediction is evaluated separately.

Q4. UCI event counts, predictions, and uncertainty

For NOX at least 110 mg/m3, held-out frequencies are 10/7,158 (0.1397%) in 2014 and 31/7,384 (0.4198%) in 2015. The 2011-2013 training estimates are 0.7818% (endpoint labels), 0.4645% (common labels), and 0.5494% (exact values). At target 100, test counts are 44/7,158 and 74/7,384.

Using only training common labels, 5,000 year-stratified block refits yield joint two-target intervals for Qproxy=AaQ_{\mathrm{proxy}}=\sqrt{\mathcal A_a}: 1.381 [1.286, 1.507] at target 100 and 1.827 [1.618, 2.116] at target 110. Both give model-based Help.

All four held-out common-only log-loss point estimates improve by 3.53-16.84%. With primary 168-observation-row blocks and correction across eight claims, target 110 in 2014 has simultaneous improvement in both log loss and Brier score. Tables 3 and 19 give every interval; Appendix E contains 24-, 72-, and 336-row sensitivities. Blocks count rows because the CSV lacks per-row timestamps. Training-audit precision and fixed-model held-out performance are distinct estimands.

Reviewer_8DR3OpenReview ↗

Reponse to authors

I appreciate the authors' detailed response. However, I feel the current results does not fully addressed my concerns. For Q1, the Weibull head test actually reverses the gain—which is exactly the model mismatch issue I was worried about. Pivoting back to tightly matched architectures doesn't solve the robustness problem. For Q2, the massive sample-size sweep basically concedes the flaw: needing 400,000 observations to reliably resolve the deepest target just proves the audit is impractical for realistic rare-event budgets. Overall, from theoretical viewpoint, I'm still not convinced this is much more than a standard stitching-together of Fisher and spectral tools.

Thank you for your careful follow-up. We would like to clarify the robustness result, the pilot budget, and the methodological contribution.

Q1. Robustness under misspecification.

Our indistinguishable-tail construction (Appendix B.2) shows that no finite set of lower thresholds can rule out arbitrary changes beyond the last observed threshold. Consistent with this result, the two-threshold audit does not flag the Weibull mismatch: the Weibull head matches both observed common survival probabilities, yet its target-8 working-model score is 2.846 while its deeper-tail predictions are biased. This is precisely the boundary our theory identifies: without additional tail assumptions, no information score based on common labels alone can certify the unobserved deeper tail. Our framework makes this boundary explicit.

The revision also establishes what additional observations can test: Proposition 3 proves local identifiability within the specified curvature family, and a split-pilot diagnostic reduces harmful Help decisions by 61.7% on the evaluated grid (15,051 to 5,764; Section 5, Tables 9–10). The diagnostic targets the specified curvature restriction, consistent with the impossibility result above. We state this scope explicitly in Section 6 (Discussion and Limitations).

Q2. Pilot cost and practical decisions.

Figure 2 shows that 200,000 common-label pilot observations yield Help at the deepest target in 199 of 200 independent pilots (99.5%). The pilot data are additional to the 20,000 neural-training observations.

Target 8 is intentionally the most demanding stress test; even there, the Help rate reaches 81.0% at 100,000 pilot observations. Other targets can be assessed with fewer observations: with 20,000, the audit issues Harm for targets 4 and 5 in 100% and 97% of pilots, respectively (Appendix B, Table 8). These correctly directed decisions steer the audit away from unfavorable common-only extrapolation using a smaller pilot budget.

For uncertain comparisons, Abstain is a substantive output: the audit declines to certify a precision advantage when the pilot interval is inconclusive. The study quantifies the data required for a decisive model-based assessment, making the practical cost explicit.

Methodological novelty.

The paper delivers a target-specific decision procedure for coarsened observations, extending beyond a Fisher-information ranking.

Proposition 2 provides an explicit perturbation bound that propagates errors in estimated gradients and information matrices into precision comparisons. Proposition 3 establishes when an additional threshold restores local identification for a specified tail family, while the indistinguishable-tail construction proves that unrestricted deeper-tail changes can preserve exactly the same common-label distribution.

Together, these results characterize which rare-target claims the available observations can support, which model restrictions can be tested through additional thresholds, and when further sampling of the same labels cannot resolve the question.

Thank you again for considering these clarifications.

Reviewer_5YAoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when labels from more common thresholds can improve the estimation of rare-event probabilities compared with directly observing the rare endpoint. The authors develop a Fisher-information-based framework to assess target identifiability and relative asymptotic efficiency, and propose a pointwise score to predict whether common-only learning will outperform endpoint-only learning. The framework is evaluated through synthetic experiments, neural networks, and two real-world datasets, including analyses of model misspecification and estimation uncertainty.

Critical strengths

The paper makes a useful distinction between identifiability, statistical efficiency, and realized prediction performance. The theoretical analysis is clear, and the authors provide a reasonably thorough experimental evaluation, including independent replications and explicit failure cases under model misspecification. The reporting of negative results is also appreciated.

Critical weaknesses

My main concern is the limited methodological novelty. The central efficiency criterion and identifiability condition appear to follow directly from classical asymptotic theory and Fisher-information-based optimal design. The spectral diagnostics and perturbation analysis are also relatively standard. It is unclear what new methodological insight is established beyond applying these existing tools to common-threshold extrapolation.

More importantly, the proposed audit relies heavily on the correctness of the assumed tail model. The paper itself shows that relatively mild misspecification can completely reverse the efficiency gain, yet the Fisher-based criterion does not appear capable of detecting such failures from common labels alone.

The evidence for predicting neural learning performance is also limited. The experiments use a correctly specified two-hazard structure, and the pointwise Fisher score is not derived from the actual neural network. The estimated-Fisher pilot fails its prespecified decision-coverage criterion, leaving the practical usefulness of the proposed audit uncertain.

Questions for authors
  1. What is the main methodological novelty relative to classical Fisher-information-based optimal design and parametric tail extrapolation? The efficiency ratio, range condition, and associated diagnostics seem largely obtainable from existing statistical theory. It would help to clarify which results go beyond a direct application of these tools, and what new capability the proposed framework provides.

  2. How can the audit be trusted when the tail model is misspecified? The curvature experiments suggest that a favorable efficiency ratio can coexist with substantial extrapolation bias. Can the authors evaluate whether the estimated audit can detect or account for such misspecification, rather than only demonstrating its consequences afterward? More generally, what guarantees or useful diagnostics remain when the assumed tail family is uncertain?

  3. How well does the proposed score generalize beyond the specific neural setting considered? The current experiments use a model structure aligned with the data-generating process, and the oracle Fisher score is evaluated across closely related targets and architectures. Could the authors test more flexible or misspecified models and include a structurally matched full-label estimator? This would help distinguish the benefit of common-label information from the benefit of imposing the correct tail structure. It would also be useful to clarify how much of the real-data improvement, particularly under temporal shift, can be attributed to statistical efficiency rather than extrapolation bias.

3 comments

Thank you for the thoughtful questions about novelty, tail assumptions, and learned prediction. We have clarified the theory and added independent tests addressing each point.

1. Contribution beyond classical Fisher theory

We develop a target-specific audit of common-threshold observation schemes. It checks whether the available labels identify the rare-target direction, compares model-based precision with direct endpoint estimation, and quantifies the independent pilot data needed for a decisive Help, Harm, or Abstain decision. Spectral, uncertainty, and predictive checks make the audit operational (Sections 2, 4, and 5).

The Fisher bound, range condition, and c-optimal design connection are classical foundations. We integrate them into a decision procedure for coarsened labels and rare targets.

An independent benchmark replication confirms deep-tail mean-error gains of 1.91-4.99 and positive rarity slopes of 0.841 and 0.858, both with positive lower confidence bounds (Section 4.1, Table 1). We also derive the asymptotic variance of the actual moment estimator. In a representative deep-tail cell, the efficient Fisher ratio is 31.86, the moment-estimator ratio is 22.32, and the squared empirical mean-error ratio is 23.05 (Figure 1, Appendix B.1).

2. Examining extrapolation assumptions

An added common threshold identifies the specified curvature family locally (Proposition 3), enabling a split-pilot likelihood-ratio diagnostic. In the independent diagnostic grid, gating reduces harmful Help decisions from 15,051 to 5,764 among 17,000 correlated decisions, a 61.7% reduction. At the largest evaluated pilot size, it rejects all nonzero-curvature conditions and records zero gated harmful Help decisions on this grid (Section 5, Tables 9-10).

We also test an alternative C-Weibull survival head. Its population solution exactly matches the two observed common survivals, yet its target-8 working-model score is 2.846 and mean absolute target log bias is 0.982. This directly measures how extrapolation depends on the survival head; the adjusted neural comparisons are in Section 4.3 and Appendix D.

The revision further gives an explicit observationally indistinguishable tail family:

Hδ(m)=H0(m)+δ(m−c3)+2,δ>0.H_{\delta}(m)=H_0(m)+\delta(m-c_3)_+^2,\qquad \delta>0.

It preserves all common-label probabilities through c3c_3, while changing the target probability beyond c3c_3 by the factor exp⁡[−δ(a−c3)2]\exp[-\delta(a-c_3)^2]. This establishes the scope of the third-threshold test within the stated model family (Section 5, Appendix B.2).

3. Separating structure from label information

Our separate 24-seed, five-method follow-up uses the same observations, targets, and three approximately parameter-matched capacities. The original predicted mean directions recur for all four decisive targets at every capacity. At the deepest target, a=8a=8, E-direct/C-structured mean-error ratios are 2.05, 2.21, and 2.19.

E-structured and F-structured share the two-hazard architecture; F-structured additionally uses common labels through the joint categorical likelihood. With architecture held fixed, F-structured improves over E-structured under the simultaneous comparison rule at a=8a=8 for all capacities and at a=7a=7 for the large model. Target-8 ratios are 3.05-3.44. This isolates the value of additional labels under the corresponding richer observation budget. Table 2 and Appendix D report all 75 cells and 45 paired simultaneous comparisons, with original and follow-up seeds analyzed separately.

Section 4.3 also specifies shared-network information, Iη=Ex[J(x)⊤Ic(x)J(x)]I_{\eta}=\mathbb E_x[J(x)^\top I_c(x)J(x)], while the matched experiments assess learned prediction directly.

4. Pilot calibration and temporal evaluation

Across 200 independent pilots per sample size, each with 2,000 bootstrap refits and five-target simultaneous intervals, target-8 Help increases from 17.0% (20,000 observations) to 47.5% (50,000), 81.0% (100,000), 99.5% (200,000), and 100.0% (500,000). Simultaneous empirical coverage is 97.5-99.5% under the specified data-generating model (Figure 2, Appendix B). Pilot observations are additional to neural training data.

On UCI gas-turbine data, all four held-out log-loss point estimates improve by 3.53-16.84%; target 110 in 2014 also has favorable multiplicity-adjusted log-loss and Brier intervals (Section 5, Table 3). A complementary USGS evaluation spans 16 test years and three magnitude targets, with all training audits and held-out results in Appendix E. These tests connect the information audit to observed temporal prediction.

Thank you for the detailed response and additional experiments. I appreciate the new analyses, especially the neural comparisons and pilot study. However, I still have concerns about the methodological novelty, as the main framework seems to be largely based on existing Fisher-information theory. Also, while the additional experiments help clarify the limitations under model misspecification, I am not sure they fully address how reliable the audit would be when the tail model is unknown. Overall, the rebuttal addresses some of my concerns, but I will maintain my original rating.

Thank you for your thoughtful follow-up and for recognizing the value of the additional experiments. We appreciate your careful assessment.

As clarified in the revised Introduction, our methodological contribution is a target-specific audit of coarsened observation schemes: it determines whether common labels retain the information needed for an unobserved rare target, whether an endpoint-relative precision advantage can be certified with finite pilot data, and which extrapolation assumptions can be tested through additional observable thresholds. This turns classical information-theoretic foundations into a concrete decision procedure with explicit uncertainty and empirical validation.

Thank you again for your time and consideration.

Reviewer_DtgrOpenReview ↗
Rating
3
Confidence
1
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies whether labels for relatively common events can improve the estimation of some extremely rare event probabilities compared with directly learning from rare-event indicators. The paper aims to characterize when the common-threshold labels are informative and when these labels can improve statistical efficiency, and when extrapolation fails.

Critical strengths

None

Critical weaknesses

None

Questions for authors

None

1 comment

Thank you for taking the time to review our paper. We appreciate your feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.