- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines a potential confound in threshold-based evaluation of rare-event dense prediction. In commonly used pool-and-threshold protocols, particularly max-pooled Critical Success Index (CSI), differences in forecast amplitude calibration can substantially affect the resulting contingency table even when the spatial ordering of predictions is unchanged. To diagnose this issue, the authors introduce the FreeKnob Audit, which fits an order-preserving monotone calibration map on held-out data separately for each system and then repeats the original evaluation without retraining the model. The resulting raw-to-calibrated score change is used as a sensitivity control for the extent to which a reported model comparison depends on amplitude calibration.
The empirical study is extensive. On SEVIR, the authors report that two checkpoints of the same architecture change from a -29.5% CSI difference before calibration to +5.3% afterward; across 450 pairwise comparisons among six systems, frequency-bias differences are strongly associated with the movement of CSI contrasts, with 51 sign reversals. For the published CasCast cascade-versus-backbone comparison, the CSI gap at the extreme operating point decreases from 0.1601 to 0.0339 after the control. Additional experiments investigate pooling conventions, event rarity, crowd-density estimation, semantic segmentation, alternative verification scores, and temporally separated calibration data.
Critical strengths
The strongest aspect of the paper is that it identifies and quantifies a practically important evaluation ambiguity using a remarkably simple control. The FreeKnob Audit requires no model retraining, preserves the raw pixel ordering, uses held-out calibration data, and applies the same procedure symmetrically to competing systems. The same-architecture checkpoint experiment is particularly persuasive because it demonstrates that a large apparent CSI difference can arise even when architecture itself cannot explain the gap.
The empirical analysis is also unusually thorough for an evaluation paper. Beyond the main SEVIR comparison, the authors analyze 450 pairwise contrasts, different pooling conventions and rare-event thresholds, an earlier calibration window, crowd-density estimation, semantic segmentation as a boundary case, alternative verification scores, and bootstrap uncertainty. These controls substantially strengthen the case that the reported phenomenon is not simply an artifact of one isolated model pair.
Critical weaknesses
The first concern is that the strongest interpretation of the FreeKnob control is specific to max pooling. A monotone transformation preserves pixel ordering and commutes with max pooling, but this is not generally true for sum or average aggregation: applying a nonlinear monotone map before aggregation may change the ordering of pooled regions. Therefore, the claim that the intervention changes calibration without affecting spatial discrimination is much cleaner for the primary SEVIR setting than for the crowd-counting or other non-max-pooling extensions. The generality claims should distinguish these cases more explicitly.
A second concern is the relatively limited methodological novelty and empirical scope behind the broader framing. The dependence of CSI on frequency bias, percentile matching, and forecast recalibration are well established in the verification literature, as the paper itself discusses. The novel contribution is primarily the symmetric held-out audit and its quantitative application to modern deep-learning checkpoints. Moreover, the most consequential published-comparison analysis remains centered on a relatively small set of SEVIR systems. This does not invalidate the contribution, but it makes the paper's significance depend heavily on whether the empirical audit is viewed as sufficiently broad for the general claims being made.
Questions for authors
Could the authors clarify how the statement that the FreeKnob intervention “cannot improve spatial ranking” should be interpreted outside the max-pooling setting? In particular, for the 32×32 patch-sum crowd-density experiment, a nonlinear monotone transformation applied before summation can in principle change the ordering of patch-level scores. It would be useful either to quantify how often such pooled-cell rank changes occur or to restrict the corresponding interpretation to the max-pooling setting.
Could the authors also clarify the precise inferential meaning of the calibrated residual in relation to the paper's title? The discussion correctly notes that amplitude accuracy at a physically meaningful threshold is itself a legitimate component of predictive skill and that the audit does not produce a corrected leaderboard or a direct estimate of spatial skill. Since the residual also depends on the selected transformation family, would it be more precise to characterize FreeKnob primarily as a sensitivity/identification audit rather than as fully “decoupling” calibration from predictive skill?
2 comments
Thanks for addressing my concerns. I will maintain my original score.
We appreciate the careful reading and the questions. We have added evidence and made the two changes you point toward.
R1. "the strongest interpretation of the FreeKnob control is specific to max pooling." We agree. A monotone map commutes with max pooling (and no pooling) but can reorder blocks under average/sum pooling. The revision now adds a control that fits the map after pooling, on the pooled field that CSI scores rather than on pixel amplitudes. So, "cannot change the ranking of pooled cells" holds by construction under every pooling. We report both controls, and note that under average pooling they measure different objects. This control raises the bias–gain correlation on the average-pooled cells from r=0.30 to 0.87, and across all 450 contrasts from 0.796 to 0.896 (ρ=0.860).
We also fix an error in the submission. Under 16×16 average pooling at τ=160 and 181, the pre-pooling map placed the cascade ahead of its backbone (+0.025, +0.018), when it trails at every threshold. The corrected control reduces average-pooled reversals from 14 to 5. The revision fixes the calibrated contrasts and the reversal count, which becomes 64 across the 450 contrasts (46 with both 95% intervals excluding zero), against 51 in the submission.
R2. "It would be useful either to quantify how often such pooled-cell rank changes occur or to restrict the corresponding interpretation to the max-pooling setting." We now quantify this. The identity, a positive gain and a positive shift preserve patch order exactly. With full 32×32 patches (edges cropped) and predictions clipped at zero, a shift adds the same 1024δ to every patch sum. Of the 320 selected maps, 263 are of these three kinds. The other 57 are a quantile map or a negative shift, which can reorder patches. On the 263 cells the bias–gain correlation is ρ=0.563, against 0.568 on all 320. We also fit the map after the patch sum, where no reordering is possible. This gives ρ=0.593. The crowd result does not depend on reordering. The revision reports these numbers.
R3. "the relatively limited methodological novelty and empirical scope behind the broader framing." We agree the dependence of CSI on frequency bias is established, and the paper credits it. Our contribution is the symmetric held-out audit and the measurement of how much of a published gain that dependence explains. On scope, the revision adds four independently released comparisons (DiffCast, FuXi, AIFS, PreDiff) and an out-of-sample prediction test. Therefore, the claim no longer rests on the six SEVIR arms alone.
R4. "Could the authors also clarify the precise inferential meaning of the calibrated residual in relation to the paper’s title?" We concede the calibrated residual is not an estimate of spatial skill, and we rename the method a sensitivity audit. The revised title is "A Free Knob: A Sensitivity Audit for Calibration in Threshold-Based Evaluation". What the audit measures is the share of a reported gain that an order-preserving recalibration of amplitudes can reproduce. That share cannot be evidence of better placement of extremes, because the map changes no ranking.
We would appreciate your reconsideration of the score considering these changes.