- Rating
- 2
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SIDP, a method for classifying multichannel physiological signals such as EEG and ECG. It uses a small set of labeled support examples to build class-specific probes, and these probes extract diagnostic information from a new recording. The support set is used only at inference, so the model does not need retraining. The method is evaluated on EEG and ECG datasets in both single-label and multi-label settings.
Critical strengths
-
All notation, tensor shapes, and hyperparameters are clearly defined, which makes the work highly reproducible. The notation is consistent and carefully organized, and it greatly helped me understand the core model architecture.
-
The experimental results are reported in a detailed and transparent way. The authors present both the cases where the model performs well and its limitations, so the reader can clearly see the strengths and limits of the proposed method.
-
The ablation studies are systematic. They make it possible to see how each component of the architecture contributes to the final performance.
Critical weaknesses
-
Insufficient baseline comparisons. I understand the authors’ point that the input representations differ across methods. Even so, many models could be compared from the perspective of the same classification task, regardless of input format. Direct comparisons with such models, or experiments with re-implemented models of similar architecture, seem necessary.
-
The performance gains appear to come from the input representation rather than the architecture. As the authors acknowledge, whether the margin input is used appears to have a large effect on performance. A more fundamental concern is the large gap in parameter count between SIDP and the MLP baseline. This raises questions about the efficiency and necessity of the proposed architecture. Because the models are not matched in size, the observed performance differences are hard to attribute to architectural advantages.
Questions for authors
- Figure 1 has low readability, and some of its text appears to overlap, so it should be revised. It is also placed after the Method section, which makes it hard to read alongside the corresponding text.
- From a classification-task perspective, there appear to be models whose performance could be compared regardless of input format. Could the authors provide such comparisons, or explain why they were not included?
- What is the practical benefit of using a support set? The performance gain from support-induced probes does not appear substantial. Could the authors elaborate on this point?
- The benefit of the proposed architecture appears to be significant mainly on the PTB dataset. Are there structural advantages that would let it generalize to other datasets?
- Is the per-recording normalization in Section 4.5 appropriate? Since it uses statistics from each test patient’s own recording, could it cause information leakage?
3 comments
The readability of the figure has improved. The authors also appear to have substantially strengthened the baseline comparisons under identical input conditions. I also appreciate that they added a comparison with a parameter-matched MLP, which directly addresses my concern about the difference in model size. However, I still have concerns about the fact that SIDP performs significantly worse than a simple MLP of the same size (Table 2). The authors explained the cause, but this seems to imply that the advantage of the proposed architecture does not show up in terms of accuracy.
In addition, SIDP’s advantage still does not hold on datasets other than multi-label PTB-XL (ADFTD and single-label PTB-XL), which remains a major weakness. This is because it raises the risk that the benefits of the proposed method depend on a specific dataset or data split. Overall, the authors have responded sincerely to the questions I raised.
Therefore, I have raised my Informativeness score to 3. However, I will maintain my overall rating of 2. Please let me know if I have misunderstood anything.
We thank you for the follow-up and for raising the Informativeness score. Your reading of Table 2 is correct, and we address both concerns below:
1. Accuracy against the equal-size MLP
You are right that the gap is significant. Under nested cross-validation, SIDP reaches 0.8604 AUROC and the equal-size MLP reaches 0.8705, a difference of 1.01 [0.77, 1.23] points. We agree that, on the five training labels, the advantage of SIDP is not higher accuracy. We do NOT claim it in the paper.
The advantage is a capability that the MLP does not have. The MLP learns a separate output weight vector for each of its five labels. A new diagnosis, therefore, needs new labelled data and retraining. SIDP has no label-specific parameters. It builds the classifier for each label at inference from ten labelled patients. In every fold of the same nested experiment, the trained SIDP scored 30 PTB-XL statements that were not among its training labels, with a mean AUROC of 0.715 and no retraining. The MLP cannot produce a score for these statements at all without retraining.
The one-point gap is the price of this capability. Fixed learned queries keep the SIDP encoder but learn one set of queries per label, like the MLP. With learned queries, the encoder comes within 0.20 AUROC points of the MLP. Most of the gap, therefore, comes from building each label's classifier from ten examples instead of learning it from the full training set. It does NOT come from a weaker encoder. On identical inputs, SIDP is also the strongest of the models that work from labelled examples. It scores above ProtoNet and Matching Networks with the same encoder, and above the Transformer and Set Transformer.
2. Results on ADFTD and single-label PTB-XL
We agree that this is a limitation, and the paper reports it in Section 4.8. Following two details may help to judge how specific it is to SIDP:
- On ADFTD (52 training patients), the MLP is ahead of every attention-based model, not only SIDP. Fixed queries, Transformer, Set Transformer and SIDP reach 0.4105, 0.4100, 0.4032 and 0.4137, against 0.5257 for the MLP. SIDP has the highest score of these four. The gap, therefore, appears to be shared by attention-based models on this small cohort, rather than specific to the probe mechanism.
- On single-label PTB-XL with per-channel inputs, the low score of SIDP points to training the encoder from a random initialization. In diagnostic runs where the encoder is warm-started, SIDP reaches 0.3999, level with fixed queries at 0.3911 (+0.88 [-0.25, +1.95]). With the margin input on the same dataset, SIDP is 2.04 [1.21, 2.88] points above the support MLP, while the MLP without supports is 0.96 points higher.
On the risk of dependence on one split, the multi-label PTB-XL results in Table 2 come from ten-fold nested patient cross-validation, NOT from a single split. The ordering of SIDP and the equal-size MLP is the same in all ten folds. On PTB, SIDP has the highest mean of the seven methods on the standard split, although its gains over the MLPs are not established under nested folds.
We hope this clarifies that SIDP is meant to trade a small amount of accuracy on fixed labels for labels that can be defined at inference. We are happy to answer any further questions.
We thank the reviewer for the positive comments on clarity and reproducibility. Locations refer to the revised PDF.
Q1. Readability and placement of Figure 1
We agree. Figure 1 has been redrawn at its printed size, so all labels are now about 7–8 pt and no labels overlap. The figure now sits at the start of the Method section (page 4, beside Secs. 3.1–3.2). Each block names the equation it implements.
Q2. Task-level comparisons beyond the descriptor input
We agree that the complete system should also be evaluated at the classification-task level. Table 2 (Table 3 in the submission) is now the primary overview of related methods. We also added task-level comparisons in which other models receive SIDP's input: an MLP, a wide MLP, a Transformer, a Set Transformer, a support MLP, and ProtoNet and Matching Networks with the SIDP encoder architecture (Table 1, lower block). On these identical inputs, SIDP scores above the attention-based and support-conditioned models, and the flattened MLP scores slightly above SIDP.
Q3. Practical benefit of labelled supports
The practical capability is that the diagnostic context can be changed by supplying labelled examples, without updating the trained model's parameters. Summaries of the positive versus negative supports generate the probes, and these probes determine what the retrieval from the query attends to. This capability does not, by itself, guarantee better accuracy than a fixed-label head.
Multi-label PTB-XL. The support-source experiment (Sec. 4.4, Table 3) gives direct evidence that the benefit can extend to support patients who were outside encoder training. SIDP's F1 difference from fixed queries is +0.85 [+0.12, +1.50] points with training supports and +0.72 [+0.05, +1.38] with validation-patient supports. With test cross-split supports it is +0.64 [−0.05, +1.22]: the point estimate is positive, but this interval does not establish an advantage.
Original PTB ablation. With training-patient supports, SIDP exceeds fixed learned queries by +2.14 [+0.75, +3.61] F1 points. With validation-patient and test cross-split supports, the differences are +2.07 [+0.75, +3.47] and +2.06 [+0.74, +3.44]. Under nested patient cross-validation, the estimated difference from fixed queries is +0.51 [−0.38, +1.41].
Other comparisons. On single-label PTB-XL with margin inputs, SIDP improves on the support MLP by +2.04 [+1.21, +2.88] F1 points after correction for six comparisons, while the ordinary MLP performs better.
Q4. Generalization beyond PTB and structural advantage
The structural capability is this: a target descriptor derived from the supports controls what is retrieved from the query's cross-channel latent slots. This connects the complete input-processing pipeline to evidence selection conditioned on the supports. Evaluating only fixed-label accuracy would miss this functionality, and evaluating only against pooled inputs would leave unclear where the accuracy gains come from. We therefore now report both system-level and matched-input evidence.
Generalization remains conditional:
Q5. Per-recording normalization and information leakage
Per-recording normalization is appropriate for an offline protocol: one where the complete query recording is available and its statistics are computed from it alone, without labels, other patients or later recordings. If a decision is made per window or online, using the entire recording changes the information available at prediction time. For multi-label PTB-XL, the statistics use only the ten-second record being classified.
Per-channel SIDP macro-F1 under alternative normalization methods (Table 39):
The table has no paired intervals for these within-SIDP changes, so it does not establish that the changes are statistically significant, nor that SIDP is invariant to normalization.
The margin-input sensitivity analysis (Table 26, Table 25 in the submission) is a different comparison. Its intervals for the window-only and training-only changes include zero. The submission's sentence "we conclude that there is no clear drop" has been removed.
The ethics statement now notes that per-window or online use requires window-only statistics. For the window-level PTB and ADFTD analyses, we will make window-only normalization the default in the final version and also report the recording-level and training-only alternatives.