- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes risk heads that adapt frozen time series forecasters into failure detection systems. The method learns a bounded correction to a binary risk logit using features derived from observed history, forecast summaries, and forecast endpoints. Training uses temporally graded labels together with false-positive, smoothness, and monotonicity penalties in the loss function.
Critical strengths
-
The problem formulation is clear and practically motivated. The distinction between accurate forecasting and useful failure-warning decisions is important. The proposed framework is forecaster-agnostic and is evaluated with several forecasting models, which is a valuable design choice.
-
The bounded logit correction provides a simple and interpretable constraint on how much the risk head can alter the base binary predictor.
-
The controlled objective comparison in Table 2 is useful. In particular, the ablations indicate that the false-positive, smoothness, and monotonicity terms affect the recall–false-positive trade-off in meaningful ways.
-
The paper provides code and relatively detailed reproducibility information.
Critical weaknesses
-
The paper reports results for small evaluation cohorts on several datasets: seven eligible events for NASA Battery for example. Under these conditions, differences in caught events can be sensitive to individual trajectories. However, no confidence intervals, uncertainty estimates, or results over multiple training seeds are reported. This is important for assessing the robustness of the claimed gains.
-
The standard baseline definition seems a bit unfavorable as the input features, number of trainable parameters are not the same between standard and risk heads. Could it be a version of the baseline matching the full risk head in terms of these?
-
Appendix C.1 states that the calibration procedures differ from model to baselines. These different calibration targets and policies make the Table 1 system comparison harder to interpret, because they can substantially influence results. What is the reason for these different calibration settings?
-
The paper does not report typical selected values of the warning threshold (\eta). How much do calibrated thresholds vary across datasets, backbones, and folds? Are the selected thresholds systematically conservative or permissive? Since this setting concerns high-stakes and rare-event detection, some additional interpretability elements would be valuable.
Questions for authors
see weaknesses above
5 comments
The full objective beats every existing objective on all three metrics, and all 18 of those intervals exclude zero. It is also ahead of temporal label smoothing in all 10 seeds. The penalty rows match the roles described in the paper.
The result also does not depend on small cohorts. Our response to Reviewer frWU (W3) adds 12 datasets with 11,018 events.
Table 6. Events warned in time on the three largest new datasets and on all 12 (in %).
| Dataset | Events | Standard binary head | Ours | Gain (95% CI) |
|---|---|---|---|---|
| Los Alamos system outages | 6,965 | 21.4 | 30.9 | +9.5 (6.3 to 12.7) |
| Alibaba rack incidents | 1,618 | 35.2 | 48.6 | +13.4 (9.6 to 17.1) |
| Scania truck repairs | 439 | 14.1 | 19.7 | +5.6 (2.1 to 9.1) |
| All 12 new datasets | 11,018 | 33.7 | 42.7 | +9.0 (7.2 to 10.8) |
Response to W2
We trained a standard head that receives exactly the inputs of our head and has the same number of trainable parameters. We also gave it the full 30-unit label, which removes the last difference in how the two heads are supervised.
Table 1. The standard head matched step by step to our head, averaged over the 20 pairs (in %). The baseline rows and the single-configuration row use the pooled 5% calibration level. The last row uses the calibration of the system comparison.
| Head | Inputs | Training label | Parameters | Event recall | FPR | Precision |
|---|---|---|---|---|---|---|
| Standard head | forecast summary | 10 units | 1.2K to 2.9K | 65.98 | 9.43 | 45.63 |
| Standard head with our inputs | forecast, history, physical limit | 10 units | 3.5K to 8.8K | 68.58 | 8.21 | 48.36 |
| Same, with our parameter count | forecast, history, physical limit | 10 units | about 3K | 68.21 | 8.27 | 48.13 |
| Same, with the 30-unit label (fully matched) | forecast, history, physical limit | 30 units | about 3K | 71.52 | 6.58 | 53.12 |
| Binary head inside our system, correction switched off | forecast, history, physical limit | 30 units | part of ours | 70.43 | 6.61 | 52.88 |
| Ours, single configuration of the controlled comparison | forecast, history, physical limit | graded | about 3K | 85.67 | 2.06 | 66.99 |
| Ours, system of the paper | forecast, history, physical limit | graded | about 3K | 87.35 | 1.80 | 70.91 |
Our inputs and parameter count raise the recall of the baseline by about two points, and the 30-unit label adds about three more. With the same pooled 5% calibration, our single-configuration head leads the fully matched baseline by 14.2 recall points and issues less than a third of its false warnings. The system of the paper leads it by 15.8 recall points (95% CI 10.2 to 21.5) and is ahead in all 20 pairs, so about three quarters of the gain in the system comparison remains. The same result appears inside our own system. Its starting logit is a binary head trained on the same inputs with the 30-unit label, and with the correction switched off it reaches the level of the matched baseline.
The parameter counts were close to begin with, 1.2K to 2.9K for the standard head and about 3K for ours. Table 2 varies the size of the fully matched baseline from a third of our parameter count to about three times as many.
Table 2. Event recall of the fully matched baseline at different sizes (in %).
| Parameters | tiny-N-CMAPSS | NASA Battery | XJTU-SY | FEMTO | All 20 pairs (recall / FPR / precision) |
|---|---|---|---|---|---|
| 1K | 92.67 | 51.43 | 62.67 | 72.94 | 69.93 / 7.21 / 51.04 |
| 3K | 93.67 | 54.29 | 64.00 | 74.12 | 71.52 / 6.58 / 53.12 |
| 10K | 96.33 | 54.29 | 65.33 | 76.47 | 73.11 / 6.13 / 54.57 |
| Ours, 3K | 99.00 | 77.14 | 82.67 | 90.59 | 87.35 / 1.80 / 70.91 |
The recall of the baseline rises by about 3 points over this range and stays more than 14 points below our head. A baseline somewhat smaller or larger than ours gives the same conclusion.
Table 3 matches the inputs in the other direction. It gives our head only the forecast summary, the input of the standard head, so each row compares the two heads on identical inputs and parameter counts.
Table 3. Both heads on identical inputs (event recall / FPR, in %).
| Inputs of both heads | Standard head, 30-unit label | Our head and objective | Recall gain (95% CI) | Pairs with higher recall |
|---|---|---|---|---|
| Forecast summary only | 68.72 / 8.13 | 78.31 / 2.94 | +9.59 (4.28 to 14.97) | 18 of 20 |
| Forecast, history, physical limit | 71.52 / 6.58 | 87.35 / 1.80 | +15.83 (10.21 to 21.48) | 20 of 20 |
Our head is ahead on identical inputs in both rows, with fewer false warnings. The extra inputs add 2.8 points to the standard head and 9.0 points to ours. Giving the standard head our inputs removes only a small part of the difference. They become more useful through the graded target and the penalties, which is the method of the paper.
The controlled comparison in the paper is matched in the same way, since every objective is trained on our inputs with our head. Against temporal label smoothing, the strongest existing objective, our objective has higher mean recall at each of 10 head seeds. In a separate experiment with 50 runs on tiny-N-CMAPSS, it warned at least 20 flights ahead on 72.03% of failures against 65.47% (+6.57 points, 95% CI 2.33 to 11.20).
We sincerely thank Reviewer aqs3 for the thoughtful and constructive review, and for the time given to our paper. We address each weakness below. We have incorporated these results into the revised paper and improved the main text so that it is easier to read.
Response to W1
We now report uncertainty for both result tables of the paper, as confidence intervals, a failure-by-failure test, and results over training seeds. Intervals are 95% and come from resampling whole assets 10,000 times, with each asset's predictions kept paired across both systems and all five forecasters.
Table 1. Change from the standard system to ours in the system comparison (percentage points, 95% CI).
Every interval excludes zero, including on NASA Battery with its seven failures.
Table 2. Failure-by-failure comparison at the original cutoffs of the paper, and recall gain after removing the most favorable failures (percentage points). For each failure we count how many of the five forecasters caught it.
Only 2 of the 99 failures are caught by fewer forecasters with our system, against 36 caught by more. Under equal systems, the chance of a split at least this uneven is below 10⁻⁸ (exact sign test). With the two most favorable failures removed from NASA Battery, the gain there is still 28 points.
We retrained both heads with 10 seeds, and the whole pipeline, forecasters included, with 5.
Table 3. Results over training seeds, averaged over the 20 pairs (mean ± standard deviation, in %).
Table 4. Event recall by dataset over the 10 head seeds (mean ± standard deviation, in %).
Our system is ahead in every seed on every dataset, and in at least 19 of the 20 pairs each time. The values in the paper lie within one standard deviation of the seed mean.
Table 5. Controlled objective comparison, full objective minus each alternative (percentage points, 95% CI).