- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a framework for prediction and attribution in irregular clinical time series. The goal is to enable an exact signed decomposition of the prediction logit into per-variable, per-timestep contributions while maintaining competitive predictive performance. The attribution procedure freezes evaluated attention/retention maps and propagates contributions through the network using effective-slope rules for nonlinearities and Jacobian-based propagation through normalization.
I found the overall goal interesting and important, particularly the attempt to jointly design the model architecture and attribution mechanism. The reported predictive performance and computational efficiency are promising. However, I have significant concerns about whether the released implementation matches the temporal architecture described in the paper, as well as some concerns about the fairness and interpretation of the experimental evaluation.
Critical strengths
Architecture-attribution co-design is an interesting and practically relevant direction. The paper addresses an important problem, producing clinically interpretable temporal explanations without sacrificing predictive performance or requiring expensive post-hoc attribution.
Critical weaknesses
-
The released implementation source code appears inconsistent with the temporal architecture described in the paper. It seems to me that the paper describes signal attention as aggregating the variables corresponding to the same temporal position before passing the resulting temporal sequence to the retention layers. However, the released implementation appears to reshape a variable-major tensor of shape directly into , without first exchanging the variable and temporal dimensions. Consequently, each group of tokens does not generally correspond to the variables at a common time point. Groups can contain different time points from the same variable and can cross variable boundaries. The manuscript describes this as a fixed permutation that can be inverted during attribution, but I do not think that explanation resolves the issue. The regrouping occurs before nonlinear signal attention and its reduction, so applying attention to these groups is not generally equivalent to correctly grouping the variables by time and then permuting the resulting representations. Exact reconstruction of the implemented network can still hold, but that only establishes faithfulness to the implemented computation, not equivalence to the architecture described in the paper.
-
From my reading, it seems that the experimental evidence does not yet cleanly establish the claimed architectural and attribution advantages. GARLIC is the most important architectural baseline because TRACE inherits its front end and mainly replaces its GRU with retention. However, TRACE is tuned on the current validation split using Optuna over architecture and optimization parameters, whereas GARLIC appears to use configurations from the original paper without being retuned on the TRACE split. The checkpoint-selection procedure for reproduced GARLIC is also not clearly matched. Since some TRACE–GARLIC improvements are modest, particularly on P19, it is difficult to determine how much of the gain comes from the retention-based architecture rather than differences in model selection and tuning.
-
I also have concerns about the gROAR evaluation. Removing an input cell changes both its value and its observation mask, while the model explicitly consumes the observation mask. The retrained model may therefore exploit the attribution-dependent pattern of retained/missing cells in addition to the retained measurements themselves. Moreover, treating removed observations as missing may allow the imputation mechanism to partially reconstruct their values. To me, these effects make the remove-and-retrain results harder to interpret as direct evidence that the selected measurements themselves are more faithful explanations.
Questions for authors
-
As far as I understand, the paper describes signal attention as grouping the variables at each common temporal position, while the released implementation appears to reshape directly into without first swapping the variable and temporal axes. I may be missing a technical detail, but I do not see how these operations are equivalent once nonlinear signal attention and reduction are applied. It would be helpful if the authors clarify why this reshaping can be treated as a fixed permutation, and whether the resulting sequence entering retention is guaranteed to preserve clinical time order?
-
From my reading, it seems that TRACE is tuned directly on the current validation split, whereas GARLIC uses the configurations reported in the original paper without comparable retuning on the TRACE split. It is also not clear to me whether checkpoint selection is matched across the two methods. Could the authors clarify the exact GARLIC tuning and checkpoint-selection protocol, and ideally provide a comparison where GARLIC is retuned under the same split, search budget, and model-selection procedure? Unless I have misunderstood, the substantially stronger reproduced performance of some baselines, such as RETAIN, compared with the numbers reported in the GARLIC paper suggests that tuning/protocol differences can have a large effect, which makes this controlled comparison especially important.
-
I am not fully convinced that the current analysis establishes that retention is necessary for exact attribution, rather than simply making the decomposition easier or more efficient. TRACE freezes input-dependent attention/retention maps during attribution, unless I have misunderstood the setup, a comparable frozen-gate convention might also be defined for a GRU. Could the authors clarify whether such a GRU decomposition would fail to be exact, fail to be unique, or simply be less computationally convenient?
-
One other concern is that the gROAR intervention changes both the selected value and its observation-mask entry, while the model explicitly uses the mask as an input. Therefore, it seems possible that the retrained model can exploit the attribution-induced mask pattern in addition to the retained measurements. Could the authors explain how this confound is ruled out?
-
My reading of the pre-processing is that z-score statistics are computed over the full cohort before the train/validation/test split. This may simply be a matter of presentation, but if that is correct, the normalization uses information from validation and test patients. Could the authors confirm whether this is the case, and if possible report whether recomputing the normalization statistics on the training split only materially changes the results?
5 comments
Response to Reviewer iuNm (2/3)
iuNm-3 gROAR changes both value and mask. The retrained model can exploit the attribution-induced mask pattern; imputation can reconstruct removed values.
Response. gROAR removes a measurement completely, its value and its mask, so the imputation can draw only on the measurements a ranking kept. To verify that the gap between exact attribution and random selection does not come from the attribution-induced mask pattern, we ran a control on the same selected cells that removes only the values of the others and leaves the mask untouched, so every method sees the identical mask (Appendix G).
The gap does not come from the mask. Exact attribution beats random selection in 25 of 25 seed-and-budget pairs on each dataset. On P12 the gap widens at every budget once the mask is held fixed, from +12.3 to +20.0 AUPRC at 10% and from +2.4 to +6.8 at 50%. On P19 the top 10% of the values by exact attribution restore 58.1 AUPRC of the full model's 58.9.
| P19, top 10% kept, test AUPRC | exact attribution | random selection | full input |
|---|---|---|---|
| values removed, mask kept (control) | 58.1 | 52.5 | 58.9 |
| values and mask removed (gROAR) | 45.9 | 18.0 | 58.9 |
Why gROAR removes the mask. With the mask left in place, a random 10% of the values already reaches 52.5 of 58.9. The mask carries the informative missingness (Che et al., 2018) of the original observations to every ranking, which leaves 6.4 points for any ranking to win, and exact attribution takes 5.6 of them. Under gROAR the same random selection falls to 18.0 and the rankings separate. Informative missingness is a property of the observation process, not of the attribution. Removing value and mask together is the design that tests the ranking itself. The perturbation tests without retraining, where no mask pattern can be learned, lead to the same conclusion (Appendix H).
References
- Che et al. Recurrent neural networks for multivariate time series with missing values. Scientific Reports, 2018.
Response to Reviewer iuNm (3/3)
iuNm-4 Frozen-gate GRU. Could a frozen-gate convention be defined for a GRU? Would it fail to be exact, unique, or just be less convenient?
Response. It fails to be unique. This question led to a new experiment and a sharper Proposition 1. Following the reviewer's suggestion, we defined the frozen-gate decomposition for a GRU and ran it on GARLIC's own checkpoints (Appendix J, Table 15).
It is exact. With the gates z_t and r_t frozen and tanh(a_t) written as a slope times a_t, the GRU becomes a linear recursion h_t = A_t h_{t-1} + B_t x_t + c_t, and its unrolling sums to the logit at round-off.
It is not unique. The slope depends on the reference point chosen for tanh, and this choice enters at every step between an input and the output. We compared two standard rules, the zero-reference ratio tanh(a)/a of the rescale rule (Shrikumar et al., 2017) and the local slope tanh′(a). Both add up to the logit, and they disagree on where the credit goes.
| Five seeds, whole test split | P12 GARLIC (GRU) | P12 TRACE | P19 GARLIC (GRU) | P19 TRACE |
|---|---|---|---|---|
| Completeness gap, single precision (median) | 1.3 × 10⁻⁶ | 1.4 × 10⁻⁶ | 3.5 × 10⁻⁷ | 2.8 × 10⁻⁷ |
| Rank agreement of the two rules | 0.90 | 1.00 | 0.91 | 1.00 |
| Top-10% overlap of the two rules (Jaccard) | 0.82 | 1.00 | 0.68 | 1.00 |
| Rank agreement across training seeds | 0.72 | 0.82 | 0.78 | 0.87 |
On P19 two equally valid rules give GARLIC top-10% sets with a Jaccard index of 0.68, so a clinician would be shown a different explanation depending on a choice that the model does not determine. In retention the contribution of step j to step t is the single number R[t, j]. No pointwise nonlinearity lies between time steps, so no rule has to be chosen there. Retention is therefore what makes an exact attribution unique. Proposition 1 is rewritten to state exactly this.
iuNm-5 z-score statistics over the full cohort. Report train-only normalisation.
Response. Recomputing the statistics on the training split only does not change the results meaningfully, i.e., the difference is very marginal (Appendix M, Table 17). The split, the configuration and the seeds are the same.
| TRACE, five seeds | P12 AUROC | P12 AUPRC | P19 AUROC | P19 AUPRC |
|---|---|---|---|---|
| whole-cohort statistics (Table 1) | 86.83 ± 0.90 | 58.87 ± 1.97 | 91.54 ± 0.47 | 58.93 ± 2.64 |
| training-split statistics | 86.68 ± 1.02 | 58.24 ± 2.20 | 91.56 ± 0.63 | 58.13 ± 3.15 |
All four differences are at most 0.8 points and inside the seed standard deviation. Whole-cohort statistics are the convention of the released loaders of this benchmark. GARLIC (Wang et al., 2026) z-scores the whole cohort before splitting (load_data in utils.py), and Latent ODE (Rubanova et al., 2019), mTAND (Shukla & Marlin, 2021) and Warpformer (Zhang et al., 2023) compute their statistics on the full dataset as well. Our loader follows the same practice. The convention and the train-only result are now stated in the paper.
We hope these additions address the reviewer's concerns.
References
- Wang et al. GARLIC: Graph attention-based relational learning of multivariate time series in intensive care. ICLR 2026.
- Shrikumar, Greenside and Kundaje. Learning important features through propagating activation differences. ICML 2017.
- Rubanova, Chen and Duvenaud. Latent ordinary differential equations for irregularly-sampled time series. NeurIPS 2019.
- Shukla and Marlin. Multi-time attention networks for irregularly sampled time series. ICLR 2021.
- Zhang et al. Warpformer: A multi-scale modeling approach for irregular clinical time series. KDD 2023.
Response to Reviewer iuNm (1/3)
We thank the reviewer for reading the released code line by line. The finding about the reshape was correct, and it made the paper better. Both models are fixed and retrained, the two are compared under one shared protocol, and each further question is answered with a new experiment.
iuNm-1 Released code inconsistent with the paper. The signal-attention reshape does not group the K variables of one time step; the “fixed permutation” argument does not hold; exact reconstruction only shows faithfulness to the implemented network.
Response. We confirm the reviewer's reading and fixed it. Signal attention now groups the K variables of each time step, as the paper describes. Every affected model is retrained, and every number in the revision comes from the fixed network. After the fix, token t of the sequence entering retention is built only from the K variables at time step t, so the sequence is in clinical time order. The “fixed permutation” passage is removed from the paper.
Origin. The reshape comes from GARLIC's released code (Wang et al., 2026), from which TRACE inherits its signal-attention module. In the official repository,
SignalAttentionLayer.forwardinmodel.pypermutes the tensor to[B, K, L, D]and reshapes it to[B·L, K, D]without exchanging the two axes, whereas Eq. 6 of the GARLIC paper defines the attention over the K signals of one time step. We applied the same fix to GARLIC and retrained it, so Table 1 compares two corrected networks.Exactness. We re-verified exactness on the new checkpoints over the full test splits with five seeds. The mean relative gap is 1.0 × 10⁻⁵ on P12 and 1.2 × 10⁻⁶ on P19 in single precision, and below 10⁻¹³ in double precision (Table 2, Appendix A.2). The decomposition is exact for the network that the paper describes.
iuNm-2 Unequal tuning. TRACE tuned with Optuna on the split; GARLIC uses paper configs without retuning; checkpoint selection not matched; reproduced RETAIN much stronger than in the GARLIC paper. Retune GARLIC under the same split, budget and selection.
Response. TRACE and GARLIC are now compared under one protocol, and TRACE is ahead on all six cells of Table 1. On P12 and P19 the two models share the split, the search budget and the model-selection rule (Appendices D and F).
Beyond accuracy, TRACE's attribution is exact and unique where GARLIC's is not (iuNm-4).
RETAIN. The two papers treat baselines differently. The GARLIC paper keeps each baseline at the settings of its original publication wherever these exist (its Appendix C.1). We tune every baseline on our split. RETAIN is selected among 27 configurations of hidden size, embedding size and learning rate by validation AUPRC (Appendix F, Tables 6 and 7). On P12 the two reproductions agree (AUROC 83.5 against 83.1), and our tuned RETAIN is stronger on P19 (86.2 against 78.1) and on MIMIC-III (88.9 against 82.4). A stronger baseline makes the comparison harder for TRACE, not easier.
References