acceptodds
Under review as a conference paper at ICLR 2027

A Free Knob: Decoupling Calibration and Predictive Skill in Threshold-Based Evaluation

Abstract

Many dense-prediction benchmarks evaluate rare events by pooling prediction and target over spatial blocks, thresholding each, and scoring the resulting contingency table. At a fixed rare operating point, the max-pooled Critical Success Index (CSI) confounds spatial discrimination with amplitude calibration: sharp observations promote many blocks above threshold, while attenuated predictions from squared-error-type regression leave the same blocks below it. We repurpose classical monotone calibration as a symmetric audit: a post-hoc transform is fitted on held-out data and applied separately to each system. The transform requires no retraining and cannot reverse pixel ordering, so any contrast it reproduces cannot establish improved spatial ranking. On SEVIR, two released checkpoints of one architecture differ by -29.5% in extreme-threshold CSI before the control and by +5.3% after it. Across 450 pairwise contrasts among 6 systems, the difference in pooled frequency-bias deviation is associated with how far the CSI contrast moves under the control ( [+0.753, +0.832]), and 51 contrasts reverse sign. At CasCast's published extreme-event operating point, the cascade-over-backbone CSI gap falls from 0.1601 to 0.0339, a 78.8% reduction; the remaining gap stays positive. The effect persists when the transform is fitted on a window preceding the test period, and it acts in both directions: calibration also reveals advantages hidden by a better-calibrated baseline. A rarity sweep on geostationary infrared imagery shows the relative gain from the transform growing as events become rarer, crowd-density experiments reproduce the bias-gain relationship under patch-sum pooling, and semantic segmentation marks the boundary: where frequency bias is already near one, the average change is small. The confound therefore requires both a fixed operating point and a training regime that leaves the output miscalibrated there. We recommend reporting pooled frequency bias and a symmetric held-out FreeKnob Audit, released as an installable package, alongside rare-event pool-and-threshold scores.

Primary area:
datasets and benchmarks
Keywords:
evaluation metrics, benchmark auditing, precipitation nowcasting, critical success index, frequency bias, calibration, spatial pooling, rare events, post-hoc recalibration, quantile mapping, dense prediction, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bF9vOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines a potential confound in threshold-based evaluation of rare-event dense prediction. In commonly used pool-and-threshold protocols, particularly max-pooled Critical Success Index (CSI), differences in forecast amplitude calibration can substantially affect the resulting contingency table even when the spatial ordering of predictions is unchanged. To diagnose this issue, the authors introduce the FreeKnob Audit, which fits an order-preserving monotone calibration map on held-out data separately for each system and then repeats the original evaluation without retraining the model. The resulting raw-to-calibrated score change is used as a sensitivity control for the extent to which a reported model comparison depends on amplitude calibration.

The empirical study is extensive. On SEVIR, the authors report that two checkpoints of the same architecture change from a -29.5% CSI difference before calibration to +5.3% afterward; across 450 pairwise comparisons among six systems, frequency-bias differences are strongly associated with the movement of CSI contrasts, with 51 sign reversals. For the published CasCast cascade-versus-backbone comparison, the CSI gap at the extreme operating point decreases from 0.1601 to 0.0339 after the control. Additional experiments investigate pooling conventions, event rarity, crowd-density estimation, semantic segmentation, alternative verification scores, and temporally separated calibration data.

Critical strengths

The strongest aspect of the paper is that it identifies and quantifies a practically important evaluation ambiguity using a remarkably simple control. The FreeKnob Audit requires no model retraining, preserves the raw pixel ordering, uses held-out calibration data, and applies the same procedure symmetrically to competing systems. The same-architecture checkpoint experiment is particularly persuasive because it demonstrates that a large apparent CSI difference can arise even when architecture itself cannot explain the gap.

The empirical analysis is also unusually thorough for an evaluation paper. Beyond the main SEVIR comparison, the authors analyze 450 pairwise contrasts, different pooling conventions and rare-event thresholds, an earlier calibration window, crowd-density estimation, semantic segmentation as a boundary case, alternative verification scores, and bootstrap uncertainty. These controls substantially strengthen the case that the reported phenomenon is not simply an artifact of one isolated model pair.

Critical weaknesses

The first concern is that the strongest interpretation of the FreeKnob control is specific to max pooling. A monotone transformation preserves pixel ordering and commutes with max pooling, but this is not generally true for sum or average aggregation: applying a nonlinear monotone map before aggregation may change the ordering of pooled regions. Therefore, the claim that the intervention changes calibration without affecting spatial discrimination is much cleaner for the primary SEVIR setting than for the crowd-counting or other non-max-pooling extensions. The generality claims should distinguish these cases more explicitly.

A second concern is the relatively limited methodological novelty and empirical scope behind the broader framing. The dependence of CSI on frequency bias, percentile matching, and forecast recalibration are well established in the verification literature, as the paper itself discusses. The novel contribution is primarily the symmetric held-out audit and its quantitative application to modern deep-learning checkpoints. Moreover, the most consequential published-comparison analysis remains centered on a relatively small set of SEVIR systems. This does not invalidate the contribution, but it makes the paper's significance depend heavily on whether the empirical audit is viewed as sufficiently broad for the general claims being made.

Questions for authors

Could the authors clarify how the statement that the FreeKnob intervention “cannot improve spatial ranking” should be interpreted outside the max-pooling setting? In particular, for the 32×32 patch-sum crowd-density experiment, a nonlinear monotone transformation applied before summation can in principle change the ordering of patch-level scores. It would be useful either to quantify how often such pooled-cell rank changes occur or to restrict the corresponding interpretation to the max-pooling setting.

Could the authors also clarify the precise inferential meaning of the calibrated residual in relation to the paper's title? The discussion correctly notes that amplitude accuracy at a physically meaningful threshold is itself a legitimate component of predictive skill and that the audit does not produce a corrected leaderboard or a direct estimate of spatial skill. Since the residual also depends on the selected transformation family, would it be more precise to characterize FreeKnob primarily as a sensitivity/identification audit rather than as fully “decoupling” calibration from predictive skill?

2 comments

We appreciate the careful reading and the questions. We have added evidence and made the two changes you point toward.

R1. "the strongest interpretation of the FreeKnob control is specific to max pooling." We agree. A monotone map commutes with max pooling (and no pooling) but can reorder blocks under average/sum pooling. The revision now adds a control that fits the map after pooling, on the pooled field that CSI scores rather than on pixel amplitudes. So, "cannot change the ranking of pooled cells" holds by construction under every pooling. We report both controls, and note that under average pooling they measure different objects. This control raises the bias–gain correlation on the average-pooled cells from r=0.30r = 0.30 to 0.870.87, and across all 450 contrasts from 0.7960.796 to 0.8960.896 (ρ=0.860\rho = 0.860).

We also fix an error in the submission. Under 16×1616 \times 16 average pooling at τ=160\tau = 160 and 181181, the pre-pooling map placed the cascade ahead of its backbone (+0.025+0.025, +0.018+0.018), when it trails at every threshold. The corrected control reduces average-pooled reversals from 14 to 5. The revision fixes the calibrated contrasts and the reversal count, which becomes 64 across the 450 contrasts (46 with both 95% intervals excluding zero), against 51 in the submission.

R2. "It would be useful either to quantify how often such pooled-cell rank changes occur or to restrict the corresponding interpretation to the max-pooling setting." We now quantify this. The identity, a positive gain and a positive shift preserve patch order exactly. With full 32×3232 \times 32 patches (edges cropped) and predictions clipped at zero, a shift adds the same 1024δ1024\delta to every patch sum. Of the 320 selected maps, 263 are of these three kinds. The other 57 are a quantile map or a negative shift, which can reorder patches. On the 263 cells the bias–gain correlation is ρ=0.563\rho = 0.563, against 0.5680.568 on all 320. We also fit the map after the patch sum, where no reordering is possible. This gives ρ=0.593\rho = 0.593. The crowd result does not depend on reordering. The revision reports these numbers.

R3. "the relatively limited methodological novelty and empirical scope behind the broader framing." We agree the dependence of CSI on frequency bias is established, and the paper credits it. Our contribution is the symmetric held-out audit and the measurement of how much of a published gain that dependence explains. On scope, the revision adds four independently released comparisons (DiffCast, FuXi, AIFS, PreDiff) and an out-of-sample prediction test. Therefore, the claim no longer rests on the six SEVIR arms alone.

R4. "Could the authors also clarify the precise inferential meaning of the calibrated residual in relation to the paper’s title?" We concede the calibrated residual is not an estimate of spatial skill, and we rename the method a sensitivity audit. The revised title is "A Free Knob: A Sensitivity Audit for Calibration in Threshold-Based Evaluation". What the audit measures is the share of a reported gain that an order-preserving recalibration of amplitudes can reproduce. That share cannot be evidence of better placement of extremes, because the map changes no ranking.

We would appreciate your reconsideration of the score considering these changes.

Thanks for addressing my concerns. I will maintain my original score.

Reviewer_4XfKOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates if weather models perform better becuse they locate extreme precipitation more accurately, or whether they predict higher intensity values. The authors take existing weather models and adjust their output value range to match an intensity distribution. They observe a score gap reduction by approx 80% between a deterministic model and a diffusion enhanced model.

Critical strengths

The finding that a simple calibration baseline can change an apparently large model improvement is interesting and useful. I also find the motivation to be compelling (albeit not communicated well). Extreme event metrics can be sensitive, and qualifying reported improvements on these metrics is important. This is especially important since these reported improvements can justify future work down the line.

Critical weaknesses

I find the writing of this paper to be incredibly dense and heavily written by AI tools. While this is not a disqualifier since the authors acknowledge that the prose was written by AI, and since AI-assisted writing can be helpful, I find the AI writing in this paper to be unreadable. The rhetorical language used by the authors makes no sense to me, and prevents me from understanding the core claims made in the paper. For instance, AI-writing is abundant in the abstract

 A rarity sweep on geostationary infrared imagery shows the relative gain from the transform growing as
events become rarer, crowd-density experiments reproduce the bias–gain relationship under patch-sum pooling, and semantic segmentation marks the boundary: where frequency bias is already near one, the average change is small. The confound therefore requires both a fixed operating point and a training regime that leaves the output miscalibrated there.

I have absolutely no idea what this means.

The rest of the manuscript is littered with these dense phrases of AI writing. "This is a different stability question from ours: stability of a post-processed potential score across threshold emphasis does not test whether an original fixed-threshold model contrast changes after symmetric marginal calibration," "Sharpness as a reported virtue," "Under no pooling this costs the forecast the pixels between its attenuated peak and the threshold," "It tests the predictive form of the claim: distortion should be ordered by the pair’s calibration gap," “The audit is an identification device.” … “a monotone marginal map rules out only what such a map can explain,” "One architecture, two released checkpoints," "A published gain: cascade against its backbone."

Next, focusing on content, I do not believe the proposed audit is novel or interesting in this iteration of the paper. The authors acknowledge that CSI's sensitivity conditions are clearly established. Empirically applying these for selected deep learning comparisons is not a significant contribution for ICLR. Furthermore, I find the evidence to be too narrow to establish any broader significance. the 450 comparisons the authors perform is largely done from two checkpoints they training sharing a same architecture. Their diffusion "cascade" is built from one backbone. The variation is mostly in thresholding and pooling, and I believe adding a greater diversity of methods would add more value to the paper (making the audit broader).

I recommend rejection. I encourage the authors to re-write their manuscript completely and re-submit to another venue.

Questions for authors

None

2 comments

We thank the reviewer for their time. We first correct a factual point that affects the assessment. We then address prose, breadth, and novelty.

R1. "the 450 comparisons the authors perform is largely done from two checkpoints they training sharing a same architecture." This is a misreading of the design. The 450 contrasts come from six systems. None of them were ours or trained by us (two non-learned baselines, persistence and pysteps, plus released checkpoints from their own authors: EarthFormer, the CasCast backbone, the CasCast cascade, and one unguided diagnostic arm). These arms gave us 15 pairs ×\times 30 cells.

R2. "I find the writing of this paper to be incredibly dense and heavily written by AI tools." We agree the abstract and introduction were too dense, and we have rewritten both in plain language. We have replaced stacked technical phrasing with short, direct sentences, and simplified the figure titles. Every passage quoted in the review has been rewritten. The rewritten abstract and introduction are in the revised PDF.

R3. "I find the evidence to be too narrow to establish any broader significance." The revision adds four independently released comparisons we did not train. We have added two further SEVIR systems (DiffCast vs PhyDNet, PreDiff vs persistence), and two settings beyond SEVIR entirely (FuXi vs IFS HRES on WeatherBench 2, AIFS vs IFS HRES on ECMWF operational output). These two settings cover five and four systems, over 360 and 270 contrasts (setup in Appendix M). The comparisons now span radar nowcasting and global NWP, alongside a held-out prediction test. Calibration removes 73–97% of the DiffCast CSI difference at the rarest thresholds. The AIFS lead widens, and the PreDiff gain does not change. The effect is not specific to one architecture or dataset.

R4. "I do not believe the proposed audit is novel or interesting in this iteration of the paper." We agree the sensitivity is established, and the paper states it. The contribution is a symmetric, held-out control that quantifies how much of a specific published comparison that sensitivity accounts for. To our knowledge, the verification literature describes this qualitatively but does not measure it on modern deep-learning checkpoints. We have also narrowed the claim. The method is now a sensitivity audit, and we no longer read the calibrated residual as spatial skill. The revised title is "A Free Knob: A Sensitivity Audit for Calibration in Threshold-Based Evaluation".

We hope the corrected scope, the plain-language rewrite, and the broader evidence address these concerns, and we would be grateful for a reconsideration of the score.

Reviewer_4XfKOpenReview ↗

Reviewer 4XfK Response to Authors

Thank you for the attempt to address points made in my review. I will maintain my score since I believe the presentation and writing fixes would require a substantial revision to the manuscript.

Reviewer_P5TBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper shows that CSI can be heavily affected by calibration rather than just spatial prediction quality. The proposed FreeKnob Audit applies a monotone calibration to each model and checks how their performance differences change. On SEVIR, some model rankings even reverse after calibration.

Critical strengths
  • The CasCast example is convincing. Its CSI advantage over the backbone drops from 0.1601 to 0.0339 after calibration. That's a pretty big difference without changing the model's spatial ranking.

  • The authors also test calibration using an earlier time window, rather than only the held-out test events. The main conclusion still holds, which makes the result more convincing.

Critical weaknesses
  • The remaining CasCast advantage changes from +12.9% under global calibration to -5.5% under per-cell calibration. This makes the interpretation quite dependent on which calibration setting is chosen.

  • The 450 comparisons come from only six systems across different evaluation settings. It would be interesting to see whether the same effect is common across more independently developed models.

Questions for authors

NA

1 comment

Thank you for the positive assessment and for highlighting the CasCast decomposition and the earlier-window check. We address the two weaknesses below.

R1. "This makes the interpretation quite dependent on which calibration setting is chosen." We agree, and we have changed what the paper reports. What we treat as the result now is the share of the gain the control removes. This share is at least 88% under all three calibration settings (88–106% for the CasCast cascade over its backbone at τ=219\tau = 219). The residual’s sign depends on the admissible calibration budget, so we no longer read spatial skill into the residual. The revision reports all three controls together.

Per-control detail (τ=219\tau = 219, CasCast cascade vs backbone):

ControlShare of the relative gain removedCalibrated residual
Global quantile map88%+12.9+12.9%
Per-cell105%−5.5-5.5%
Exact (after-pooling)106%−6.5-6.5%

R2. "It would be interesting to see whether the same effect is common across more independently developed models." We agree. The intervals in the submission condition on the six SEVIR arms. To show the effect is not specific to them, the revision adds four independently released comparisons:

ComparisonQuantityBefore calibrationAfter calibration
DiffCast vs PhyDNetthreshold-averaged relative gain, 16×1616 \times 16 max pooling+82.9+82.9%+5.0+5.0%
FuXi vs ECMWF IFS HRESCSI difference, 72 h, 50 mm−0.025-0.025+0.082+0.082
ECMWF AIFS vs ECMWF IFS HRESCSI difference, 72 h, 50 mm+0.050+0.050+0.085+0.085
PreDiff vs persistencethreshold-averaged relative gain, no pooling+34.1+34.1%+34.2+34.2%

The DiffCast gain almost disappears. At its rarest thresholds (τ≥133\tau \geq 133), calibration removes 73–97% of the CSI difference. The FuXi difference reverses sign, and the AIFS lead widens. PreDiff and persistence are both close to unbiased, so the audit predicts no change. The PreDiff gain stays where it was, and no separated cell loses more than 4% of its contrast.

We also add an out-of-sample test. We fit a line on the six SEVIR systems, with intercept 0.00150.0015 and slope 0.0860.086 on the calibration gap. The line predicts the DiffCast and PreDiff movements. For PreDiff, all 29 cells fall inside the 95% band, with mean absolute error 0.003 CSI. For DiffCast, the rank correlation is 0.92 over 84 contrasts, and 81% of the 59 contrasts in the fitted range fall inside the band. The effect can be predicted from the biases alone.

We hope these additions strengthen your confidence in the generality of the finding, and we would appreciate your reconsideration of the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.