- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks when common-threshold labels can estimate a rare-event probability more accurately than observing the endpoint directly, at the same sample size. The authors propose a target-specific audit using Fisher-range identifiability, an endpoint-to-common asymptotic variance ratio, and diagnostics for estimated information. A two-rate benchmark shows deep-tail gains as well as failures caused by missing information and tail misspecification. In a prospective neural experiment, the Fisher score correctly predicts the direction of the mean error difference for four targets across three model sizes, while an independent pilot abstains on the two deepest targets. Results are mixed on USGS earthquake data, while common-only extrapolation achieves lower test log loss on gas-turbine data.
Critical strengths
- The paper separates whether common-threshold labels identify a rare target from whether they offer an efficiency advantage over endpoint observations. The Fisher-range condition and endpoint-relative variance ratio provide a concrete framework for this distinction, while the help/harm/abstain rule accounts for uncertainty in estimated information.
- Removing a threshold makes the target unidentifiable, while tail curvature introduces enough bias to reverse the common-only gain. Adding another threshold restores identifiability, but does not necessarily give an efficiency advantage.
- The authors provide a separate replication, bootstrap checks, and detailed protocols. They also report the failed near-crossover replication cell and the pilot's failure to meet its decision-coverage criterion.
Critical weaknesses
- The neural experiment is closely matched to the data-generating model. The DGP, , and common-only network all use the same two-hazard structure, with only one context variable. The 12/12 directional matches also come from just four target-level predictions evaluated at three network widths, rather than 12 independent tests. Moreover, only the large-model cell at a=8 has a positive bootstrap lower bound for the paired error difference. It remains unclear whether predicts gains with a different model or survival head.
- Much of the theory builds on established results. Theorem 1 draws on local asymptotic efficiency and c-optimal design, while Propositions 1–2 use standard spectral and matrix perturbation arguments. I see the main contribution in bringing these tools together into a target-specific audit, rather than in the individual theoretical results.
- The pilot does not resolve the strongest positive case. At a=8, where the neural experiment shows its largest common-only gain, the pilot estimates with an interval of , compared with the oracle value of 1.43, and abstains. It also fails the pre-specified decision-coverage criterion. Abstention is reasonable here, but it is unclear how much pilot data would be needed to make a reliable positive decision.
- The real-data experiments provide only limited validation of the full audit procedure. On USGS data, common-only estimation performs better in only 6, 6, and 7 of 16 yearly folds across the three targets. The gas-turbine results are more positive, but the experiment was designed after data inspection and covers only two test years. Although the fitted training-set efficiency ratios exceed one, the study does not report uncertainty bounds or pre-specified help/harm/abstain decisions. Temporal shifts also make it difficult to separate efficiency gains from changes in prediction bias.
Questions for authors
Q1. Could the authors repeat the Section 4.3 experiment with a different common-only survival head that does not match the DGP's two-hazard structure, while still allowing extrapolation from common labels? Keeping the data, targets, and model capacity comparable would make it easier to see whether remains predictive under model mismatch.
Q2. Across repeated independent pilots, how often would the lower confidence bound for exceed 1.05? A sample-size sweep reporting this positive-decision rate and interval coverage would help assess how much data the audit needs.
Q3. Could the authors check Eq. (3) against Table 2? The equation takes the square root of a ratio of averaged variances, whereas the primary metric averages absolute log errors across contexts. Under a pointwise Gaussian approximation, averaging pointwise standard deviations seems more directly related to this metric. Using the stated DGP, I obtain from Eq. (3), but approximately 1.43 when averaging pointwise standard deviations, matching Table 2. Which aggregation was implemented, and does this choice affect the pilot decisions?
Q4. In the gas-turbine experiment, the training endpoint and common-only estimates at target 110 are 0.00782 and 0.00464. What are the observed exceedance frequencies in 2014 and 2015? Could the authors also report the audit's help/harm/abstain decisions, with uncertainty, using only the 2011–2013 common-label data? This may help separate an efficiency gain from improved calibration under temporal shift.
3 comments
Reponse to authors
I appreciate the authors' detailed response. However, I feel the current results does not fully addressed my concerns. For Q1, the Weibull head test actually reverses the gain—which is exactly the model mismatch issue I was worried about. Pivoting back to tightly matched architectures doesn't solve the robustness problem. For Q2, the massive sample-size sweep basically concedes the flaw: needing 400,000 observations to reliably resolve the deepest target just proves the audit is impractical for realistic rare-event budgets. Overall, from theoretical viewpoint, I'm still not convinced this is much more than a standard stitching-together of Fisher and spectral tools.
Thank you for your careful follow-up. We would like to clarify the robustness result, the pilot budget, and the methodological contribution.
Q1. Robustness under misspecification.
Our indistinguishable-tail construction (Appendix B.2) shows that no finite set of lower thresholds can rule out arbitrary changes beyond the last observed threshold. Consistent with this result, the two-threshold audit does not flag the Weibull mismatch: the Weibull head matches both observed common survival probabilities, yet its target-8 working-model score is 2.846 while its deeper-tail predictions are biased. This is precisely the boundary our theory identifies: without additional tail assumptions, no information score based on common labels alone can certify the unobserved deeper tail. Our framework makes this boundary explicit.
The revision also establishes what additional observations can test: Proposition 3 proves local identifiability within the specified curvature family, and a split-pilot diagnostic reduces harmful Help decisions by 61.7% on the evaluated grid (15,051 to 5,764; Section 5, Tables 9–10). The diagnostic targets the specified curvature restriction, consistent with the impossibility result above. We state this scope explicitly in Section 6 (Discussion and Limitations).
Q2. Pilot cost and practical decisions.
Figure 2 shows that 200,000 common-label pilot observations yield Help at the deepest target in 199 of 200 independent pilots (99.5%). The pilot data are additional to the 20,000 neural-training observations.
Target 8 is intentionally the most demanding stress test; even there, the Help rate reaches 81.0% at 100,000 pilot observations. Other targets can be assessed with fewer observations: with 20,000, the audit issues Harm for targets 4 and 5 in 100% and 97% of pilots, respectively (Appendix B, Table 8). These correctly directed decisions steer the audit away from unfavorable common-only extrapolation using a smaller pilot budget.
For uncertain comparisons, Abstain is a substantive output: the audit declines to certify a precision advantage when the pilot interval is inconclusive. The study quantifies the data required for a decisive model-based assessment, making the practical cost explicit.
Methodological novelty.
The paper delivers a target-specific decision procedure for coarsened observations, extending beyond a Fisher-information ranking.
Proposition 2 provides an explicit perturbation bound that propagates errors in estimated gradients and information matrices into precision comparisons. Proposition 3 establishes when an additional threshold restores local identification for a specified tail family, while the indistinguishable-tail construction proves that unrestricted deeper-tail changes can preserve exactly the same common-label distribution.
Together, these results characterize which rare-target claims the available observations can support, which model restrictions can be tested through additional thresholds, and when further sampling of the same labels cannot resolve the question.
Thank you again for considering these clarifications.
Thank you for the specific suggestions on alternative survival heads, independent pilots, aggregation, and temporal event frequencies. We added the requested checks and report their statistical interpretation below.
Q1. Alternative head and matched controls
We ran a separate 24-seed, five-method study using 20,000 training and 5,000 validation observations per seed, the same targets, and three approximately parameter-matched capacities. E-direct uses endpoint labels; C-structured uses common labels with a two-hazard head; C-Weibull uses the same common-label likelihood with a Weibull head; E-structured uses the two-hazard architecture with endpoint supervision; and F-structured uses that architecture with joint common-plus-endpoint supervision.
The original common-only mean directions recur for all four decisive targets at every capacity. At a=8, E-direct/C-structured mean-error ratios are 2.05, 2.21, and 2.19. Holding architecture fixed, F-structured improves over E-structured under the simultaneous comparison rule at a=8 for all sizes and at a=7 for the large model. Its target-8 ratios are 3.05-3.44. This quantifies the value of additional common-label information alongside endpoint labels.
The Weibull control distinguishes common-label fit from target extrapolation. Its population solution fits both common survivals exactly but has target-8 working-model score 2.846 and mean absolute log bias 0.982. Adjusted comparisons favor E-direct at targets 4-7; those at target 8 are inconclusive. Section 4.3 and Appendix D give all 75 method-capacity-target cells, 45 paired simultaneous comparisons, and the alternative-head specification. Original and follow-up seed cohorts are kept separate.
Q2. Independent pilots and sample requirements
For each of five sample sizes, we generated 200 independent pilots and used 2,000 row-bootstrap refits per pilot for simultaneous 95% intervals across five targets. Help requires the lower endpoint to exceed 1.05. At target 8:
Thus 199/200 pilots resolve model-based Help at the deepest target with 200,000 observations. Figure 2 and Appendix B report all targetwise decisions, joint coverage, the original pilot outcome, and the 5,000-refit sensitivity. Pilot data are additional to the neural training sample.
Q3. Equation (3) and same-draw comparison
Equation (3), its implementation, and Table 2A use the ratio of averaged pointwise standard deviations:
QSD=⟨Vc(x)⟩x⟨Ve(x)⟩x.
For regular pointwise estimators under the stated Gaussian and absolute-moment conditions, expected absolute log error is proportional to SD, motivating this aggregation across contexts. At a=8, the oracle value is 1.434293 (Table 2A: 1.43); the distinct RMS ratio QRMS=⟨Ve⟩x/⟨Vc⟩x is 1.476253.
We replayed both on the same 1,000 original bootstrap-fitted coefficient vectors. At target 8, SD gives 0.990560 [0.588590, 1.762004], while RMS gives 1.004206 [0.589879, 1.877100]. All five original audit decisions remain unchanged. Appendix D, Table 12 reports every target. Shared-network prediction is evaluated separately.
Q4. UCI event counts, predictions, and uncertainty
For NOX at least 110 mg/m3, held-out frequencies are 10/7,158 (0.1397%) in 2014 and 31/7,384 (0.4198%) in 2015. The 2011-2013 training estimates are 0.7818% (endpoint labels), 0.4645% (common labels), and 0.5494% (exact values). At target 100, test counts are 44/7,158 and 74/7,384.
Using only training common labels, 5,000 year-stratified block refits yield joint two-target intervals for Qproxy=Aa: 1.381 [1.286, 1.507] at target 100 and 1.827 [1.618, 2.116] at target 110. Both give model-based Help.
All four held-out common-only log-loss point estimates improve by 3.53-16.84%. With primary 168-observation-row blocks and correction across eight claims, target 110 in 2014 has simultaneous improvement in both log loss and Brier score. Tables 3 and 19 give every interval; Appendix E contains 24-, 72-, and 336-row sensitivities. Blocks count rows because the CSV lacks per-row timestamps. Training-audit precision and fixed-model held-out performance are distinct estimands.