- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper predicts neonatal acidaemia (umbilical-artery pH < 7.15) from the last 30 minutes of intrapartum CTG (fetal heart rate and uterine contractions, sampled at 4 Hz). It uses the public CTU-UHB database (551 records, 105 positives). The proposed model, FORGE-CTG, is a small patch transformer (about 0.7M parameters) with two additions: 1. SCOS: four learnable "outcome queries" (adapted from FORMED) that predict the binary risk, four nested ordinal pH thresholds (7.20/7.15/7.10/7.05; CORAL-style), the continuous pH (Gaussian NLL), and, when available, the base deficit. 2. QVD: a mean-teacher consistency loss. An EMA teacher sees an FHR trace in which gaps of up to 120 s are linearly interpolated and jumps are median-filtered, while the student sees the raw trace. Hand-crafted reliability features (flatline, jumps, range, and missing runs) are added as patch statistics.
Critical strengths
- Clinically motivated ideas. Using the continuous pH and base deficit instead of a single cut-off, and treating signal loss as a quality condition, are sensible choices for CTG.
- Broad set of baselines. Clinical-feature models, MiniRocket, many neural time-series models, missingness-oriented models, and pretrained CTG encoders are compared using shared record-level folds.
- Additional analyses. Missingness stress tests, label-efficiency experiments, signal-defined subgroups, and ECG experiments give a more complete picture than accuracy alone.
Critical weaknesses
W1. The method combines known techniques, and the gain over simple baselines is small and not significant.
- SCOS is multi-task learning built from standard components: weighted BCE, CORAL ordinal regression, Gaussian NLL regression, an auxiliary base-deficit head, and the FORMED query interface. • QVD is a mean teacher with an "easier" teacher view. It is closely related to weak/strong-view consistency (e.g., FixMatch) and to learning with privileged information or generalized distillation, neither of which is cited. Also, the teacher is an EMA of the student, not a separately trained model, so calling the method "distillation" is a stretch. • Not better than simple models. FORGE-CTG is not significantly better than 10 of the 18 comparators, including clinical logistic regression (+0.018, CI [−0.017, 0.056]), random forest, XGBoost, a vanilla Transformer, PatchTST, and SAITS. Yet the main text mentions only three of these. All eight significant wins are against models with AUROCs between 0.49 and 0.65, and several of these models appear to be broken. • So the claimed significance for clinical monitoring is not shown: a logistic regression on clinical features performs about as well.
W2. The predicted risks are poorly calibrated, even worse than a constant baseline. This contradicts the claim of "more reliable risk estimates".
- FORGE-CTG's Brier score is 0.2127 (Table 1), whereas a model that always predicts the prevalence (105/551 ≈ 0.191) has a Brier score of 0.191 × 0.809 ≈ 0.154. So FORGE-CTG's probabilities are less accurate than a constant prediction, and its ECE is 0.26 (Table 3). The likely cause is the positive-class weight (about 4.2×) in the BCE loss, which pushes all predicted risks upward.
- At 90% sensitivity, specificity is only 27.8%, lower than that of SAITS (35.4%), PatchTST (29.8%), InceptionTime (30.3%), and clinical LR (28.3%). In other words, about 72% of non-acidaemic births would be flagged.
- The abstract's conclusion that the approach "improve[s] the reliability of risk estimates" is therefore not supported. Recalibration (Platt scaling or isotonic regression, fitted on validation folds) and a decision-curve analysis are needed.
W3. Many baselines appear broken or untuned.
- Several standard models perform at or near chance: GRU-D 0.4914, FORMED 0.5029, LSTM 0.5177, TimesNet 0.5230, GRU 0.5414, and iTransformer 0.5587.
- The paper notes that in its GRU-D implementation "an all-observed mask leaves input imputation inactive" (App. D), so GRU-D's key mechanism was in effect switched off. On PTB-XL, the same GRU-D reaches 0.9041, higher than FORGE-CTG (0.8879). This suggests that the CTG adaptation (7,200-step sequences) failed, not that the method is weak.
- The pretrained encoders (CTGDL-PatchTST, FORMED) are frozen and not fine-tuned.
- The baselines are single-seed runs whose tuning is not described, whereas FORGE-CTG is a 5-seed ensemble (and seed averaging usually raises AUROC).
- Established CTG-specific models are missing: feature-based CTU-UHB classifiers (e.g., Spilka et al., IEEE JBHI 2017) and CTG CNNs (e.g., Petrozziello et al., IEEE Access 2019).
W4. The clinical setup is narrow and partly unrealistic. • The paper uses only the final 30 minutes before delivery. This requires knowing the time of delivery, which is future information during labor, and leaves almost no time to act. It also mixes first- and second-stage traces. Results for earlier windows (e.g., 60–90 min before delivery, or the first stage only) are needed to judge clinical usefulness. • The study uses a single small retrospective cohort (n = 551) and has no external validation, as the authors acknowledge. • The clinical LR baseline is never specified (which features does it use?). This matters, because it nearly matches the model. • The paper states that record 1500 has no signal file, so only 551 records are used. However, the public CTU-UHB release lists 552 records. Please confirm whether this is a local data problem.
W5. The subgroup analysis does not show a benefit. • The gap between the pooled AUROC and that of the worst cluster is larger for FORGE-CTG at k = 2 (0.020 vs. 0.004) and k = 4 (0.170 vs. 0.166), and smaller only at k = 3. • The worst cluster at k = 4 (36 records) reaches an AUROC of only 0.558. • The paper does not describe how the clusters were built (features, algorithm) or what the "reference backbone" is.
Questions for authors
Please see Weaknesses above.
2 comments
Revision_v3: Updated Baseline Comparisons, Risk Calibration, Prediction Timing, and Subgroup Analyses
Your prevalence calculation and clinical logistic-regression comparison identify the right checks for this task. We used them to examine how well the model orders records, how accurately its probabilities estimate event frequency, and how early the signal is available.
- Contribution and comparison with a simple model (W1). SCOS uses the FORMED query interface to predict four related metabolic outcomes. QVD trains the raw-input CTG encoder against a weight-averaged teacher that sees a bounded FHR repair. TFC aligns the tokens used by all four queries across two raw student passes. We cite FORMED, CORAL, Mean Teacher, FixMatch, generalized distillation and R-Drop where those ideas enter the design. The word “distillation” in QVD refers to the student’s feature target from the EMA teacher.
Across the same 551 records, the complete model reaches 0.7329 AUROC against 0.6956 for the pooled binary model (exploratory paired difference +0.0373, 95% CI 0.0071 to 0.0657). Adding SCOS gives the largest individual increment, +0.0302 (nominal CI 0.0063 to 0.0547, Holm p=0.060 over five comparisons). Against a five-seed clinical LR, the full model gains +0.0144 AUROC (95% CI −0.0224 to 0.0514). This cohort leaves their relative performance uncertain.
- Probability calibration and practical operating points (W2). You were right that the original Brier score of 0.2127 exceeded the pooled prevalence reference of 0.1542. The revised CTG trainer uses unweighted binary BCE. On the saved five-seed predictions, the full model has Brier 0.1396 and ECE 0.0439, compared with 0.1436 and 0.0164 for the binary arm. The improvement in squared probability error therefore coexists with a higher calibration error under ten-bin ECE. The change in class weighting also affects the comparison with the submitted model.
For the separate five-seed validation-fitted calibration analysis, raw, Platt and isotonic Brier scores were 0.1389, 0.1385 and 0.1496. The reliability plot and decision curves show where net benefit changes with the threshold. In the earlier operating-point experiment, a validation-selected threshold of 0.125 produced held-out sensitivity 0.905, specificity 0.381 and PPV 0.256. At that sensitivity about 62% of negatives would be flagged. The clinical burden you identified persists. Figure 3 and Appendix F give the curves and operating-point results.
-
Baseline implementation and tuning (W3). Table 1 cites the original comparator models and labels their seed budgets. For eight available reruns we matched the five model seeds and record folds. Baseline hyperparameters were selected on validation records, with model-specific grids and a 40-epoch cap. We supplied the FHR mask and elapsed-time inputs to GRU-D. Its rerun still yields 0.4975 AUROC, so that row describes the performance of our CTG adaptation. The CTGDL-PatchTST and FORMED rows used frozen pretrained encoders. We cite Spilka et al. and Petrozziello et al. as CTG-specific comparators in the literature and identify the clinical-feature LR as our own implementation.
-
Clinical LR, timing and cohort (W4). The LR uses 19 FHR/UC features covering baseline rate, variability, accelerations and decelerations, threshold durations, contractions and signal loss. Scaling uses training-fold statistics and validation selects the regularization. The appendix gives the full list, including two short-term-variability columns that compute the same quantity. The public CTU-UHB release contains record 1500’s signal file. Its absence was local, and the original folds cover the other 551 records (105 positive).
The input ends at the last available sample. In this cohort, the headers place that sample at delivery. In the historical timing analysis on 505 records (92 positive), the delivery-anchored model reached 0.7484 AUROC. Using the first-stage window gave 0.5617 for the model and 0.6259 for clinical LR. Earlier landmark AUROCs were 0.5683 at 30 minutes (531 records) and 0.4286 at 60 minutes (37 records). No eligible recording reached the 90-minute landmark. We restrict the intended use of these results to retrospective near-delivery ranking. External CTG validation remains open.
- Worst-cluster performance (W5). We standardized six FHR summaries and four UC summaries and fitted K-means to all 551 records without outcomes (k=2, 3, 4, ten initializations, seed 0). The earlier reference backbone concatenated FHR and UC tokens used masked-reconstruction pretraining on the training fold, and froze the encoder for the outcome readout. We evaluated both sets of predictions on the same post-hoc clusters. The pooled-minus-worst AUROC gap narrowed only at k=3. At k=4, the submitted model’s worst cluster had 36 records and AUROC 0.5579. Appendix F reports these signal-defined subgroups, their sizes and the same-cluster reference results. We withdraw the general subgroup-robustness claim.
Updated Baseline Comparisons, Risk Calibration, Prediction Timing, and Subgroup Analyses
W1: Contribution and baseline comparison:
The revision defines DPC as a symmetric scalar-risk penalty between two independent dropout passes and relates it to R-Drop. The related-work section also situates training-view consistency alongside FixMatch and generalized distillation. For precision, we describe the raw/repaired mechanism as EMA-teacher feature consistency: the teacher is an exponential moving average of the student, not a separately trained network. The matched ten-seed DPC estimate is +0.0110 AUROC (95% CI [0.0007, 0.0228], raw p=0.038, Holm-adjusted p=0.076); the prespecified five-seed estimate is +0.0167 (95% CI [0.0021, 0.0318], Holm-adjusted p=0.038). The single-head result, 0.7299, is reported separately from these reference-pipeline contrasts. The main benchmark reruns eight baselines on shared record folds and uses paired intervals. The strongest clinical comparator is logistic regression at 0.7139; its paired difference from DPC is +0.0259 (95% CI [-0.0067, 0.0601], p=0.108).
W2: Calibration and operating point:
The revised evaluation reports probability metrics alongside ranking. For ten-seed out-of-fold predictions, DPC changes Brier from 0.1372 to 0.1357 and ten-bin ECE from 0.0248 to 0.0233. The Brier difference is -0.0015 (95% CI [-0.0031, -0.0001]); the ECE interval spans zero. The prevalence-only Brier reference is 0.1542. A threshold of 0.125 selected on validation records for a 90% sensitivity target yields held-out sensitivity 0.905, specificity 0.381, and PPV 0.256. The revised paper also includes a nested calibration audit and decision-curve analysis, with model and seed protocol identified for each result.
W3: Baseline implementation and tuning:
The main benchmark uses eight rerun baselines, each evaluated on the shared five record folds with ten fixed seeds. Trainable baselines use the same three-candidate validation-only selection, a 40-epoch cap, and early stopping. GRU-D receives the validity mask and elapsed-time input. The clinical logistic-regression model uses 19 record-level FHR/UC features with training-fold scaling and validation-fold regularization selection.
W4: Prediction timing and cohort count:
The CTU-UHB release lists 552 recordings. Record 1500 has no signal file, leaving 551 usable records, including 105 with pH < 7.15. Signal-end time equals delivery time in all 552 headers. The revised analysis separates the delivery-anchored window from first-stage-only and earlier landmark windows. The first-stage window is the 30 minutes ending at the annotated stage boundary; on its 505 records (92 positive), DPC AUROC is 0.5617, compared with 0.6259 for clinical logistic regression and 0.7484 for delivery-anchored DPC on the same records. The landmark analysis gives AUROC 0.5683 at 30 minutes earlier (531 records), 0.4286 at 60 minutes earlier (37 records), and no eligible records at 90 minutes. Stage duration defines the retrospective window and is not a model input.
W5: Signal-defined subgroup analysis:
The appendix now specifies the ten label-free FHR/UC statistics, standardisation, and K-means settings (𝑘 = 2, 3, 4, ten initialisations, random state 0). For the five-seed base model, the worst-cluster AUROCs are 0.7080, 0.6740, and 0.5579, with pooled-minus-worst-cluster gaps of 0.0203, 0.0543, and 0.1704, respectively. Cluster sizes are reported, including 36 records in the smallest 𝑘 = 4 cluster.