acceptodds
Under review as a conference paper at ICLR 2027

TRACE: Temporal Relevance Attribution for Clinical Explanations

Abstract

Irregularly sampled multivariate clinical time series are central to intensive care unit (ICU) risk prediction, yet deployable models must be both highly accurate and faithfully interpretable. Practitioners currently face a sharp trade-off: inherently interpretable architectures often rely on heuristic attention mechanisms without strict faithfulness guarantees, while rigorous post-hoc methods such as Integrated Gradients incur prohibitive computational costs. We introduce Temporal Relevance Attribution for Clinical Explanations (TRACE), a framework that resolves this tension through architecture-attribution co-design, achieving the strongest predictive performance together with single-pass, mathematically exact attribution. TRACE structurally reduces to an affine map for any fixed input, enabling a single backward pass to exactly decompose the prediction logit into signed per-variable, per-timestep contributions with a completeness gap at floating-point round-off. On PhysioNet-2012, PhysioNet-2019, and MIMIC-III, TRACE achieves the highest AUROC and AUPRC of all baselines. Under a graded remove-and-retrain evaluation (gROAR) that sweeps retention budgets with trend and equivalence testing, TRACE consistently outperforms existing inherently interpretable models. Its top- degradation is comparable to that of Integrated Gradients, yet it runs two orders of magnitude faster and preserves machine-precision faithfulness, with a relative completeness gap on the order of . Together, these results show that predictive accuracy, attribution faithfulness, and computational efficiency need not be traded off in clinical time-series modeling.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, explainable AI, feature attribution, clinical time series, irregularly sampled time series, retentive networks, attribution faithfulness, remove-and-retrain evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iuNmOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a framework for prediction and attribution in irregular clinical time series. The goal is to enable an exact signed decomposition of the prediction logit into per-variable, per-timestep contributions while maintaining competitive predictive performance. The attribution procedure freezes evaluated attention/retention maps and propagates contributions through the network using effective-slope rules for nonlinearities and Jacobian-based propagation through normalization.

I found the overall goal interesting and important, particularly the attempt to jointly design the model architecture and attribution mechanism. The reported predictive performance and computational efficiency are promising. However, I have significant concerns about whether the released implementation matches the temporal architecture described in the paper, as well as some concerns about the fairness and interpretation of the experimental evaluation.

Critical strengths

Architecture-attribution co-design is an interesting and practically relevant direction. The paper addresses an important problem, producing clinically interpretable temporal explanations without sacrificing predictive performance or requiring expensive post-hoc attribution.

Critical weaknesses
  1. The released implementation source code appears inconsistent with the temporal architecture described in the paper. It seems to me that the paper describes signal attention as aggregating the KK variables corresponding to the same temporal position before passing the resulting temporal sequence to the retention layers. However, the released implementation appears to reshape a variable-major tensor of shape [B,K,T′,D][B,K,T',D] directly into [BT′,K,D][BT',K,D], without first exchanging the variable and temporal dimensions. Consequently, each group of KK tokens does not generally correspond to the KK variables at a common time point. Groups can contain different time points from the same variable and can cross variable boundaries. The manuscript describes this as a fixed permutation that can be inverted during attribution, but I do not think that explanation resolves the issue. The regrouping occurs before nonlinear signal attention and its reduction, so applying attention to these groups is not generally equivalent to correctly grouping the variables by time and then permuting the resulting representations. Exact reconstruction of the implemented network can still hold, but that only establishes faithfulness to the implemented computation, not equivalence to the architecture described in the paper.

  2. From my reading, it seems that the experimental evidence does not yet cleanly establish the claimed architectural and attribution advantages. GARLIC is the most important architectural baseline because TRACE inherits its front end and mainly replaces its GRU with retention. However, TRACE is tuned on the current validation split using Optuna over architecture and optimization parameters, whereas GARLIC appears to use configurations from the original paper without being retuned on the TRACE split. The checkpoint-selection procedure for reproduced GARLIC is also not clearly matched. Since some TRACE–GARLIC improvements are modest, particularly on P19, it is difficult to determine how much of the gain comes from the retention-based architecture rather than differences in model selection and tuning.

  3. I also have concerns about the gROAR evaluation. Removing an input cell changes both its value and its observation mask, while the model explicitly consumes the observation mask. The retrained model may therefore exploit the attribution-dependent pattern of retained/missing cells in addition to the retained measurements themselves. Moreover, treating removed observations as missing may allow the imputation mechanism to partially reconstruct their values. To me, these effects make the remove-and-retrain results harder to interpret as direct evidence that the selected measurements themselves are more faithful explanations.

Questions for authors
  1. As far as I understand, the paper describes signal attention as grouping the KK variables at each common temporal position, while the released implementation appears to reshape [B,K,T′,D][B,K,T',D] directly into [BT′,K,D][BT',K,D] without first swapping the variable and temporal axes. I may be missing a technical detail, but I do not see how these operations are equivalent once nonlinear signal attention and reduction are applied. It would be helpful if the authors clarify why this reshaping can be treated as a fixed permutation, and whether the resulting sequence entering retention is guaranteed to preserve clinical time order?

  2. From my reading, it seems that TRACE is tuned directly on the current validation split, whereas GARLIC uses the configurations reported in the original paper without comparable retuning on the TRACE split. It is also not clear to me whether checkpoint selection is matched across the two methods. Could the authors clarify the exact GARLIC tuning and checkpoint-selection protocol, and ideally provide a comparison where GARLIC is retuned under the same split, search budget, and model-selection procedure? Unless I have misunderstood, the substantially stronger reproduced performance of some baselines, such as RETAIN, compared with the numbers reported in the GARLIC paper suggests that tuning/protocol differences can have a large effect, which makes this controlled comparison especially important.

  3. I am not fully convinced that the current analysis establishes that retention is necessary for exact attribution, rather than simply making the decomposition easier or more efficient. TRACE freezes input-dependent attention/retention maps during attribution, unless I have misunderstood the setup, a comparable frozen-gate convention might also be defined for a GRU. Could the authors clarify whether such a GRU decomposition would fail to be exact, fail to be unique, or simply be less computationally convenient?

  4. One other concern is that the gROAR intervention changes both the selected value and its observation-mask entry, while the model explicitly uses the mask as an input. Therefore, it seems possible that the retrained model can exploit the attribution-induced mask pattern in addition to the retained measurements. Could the authors explain how this confound is ruled out?

  5. My reading of the pre-processing is that z-score statistics are computed over the full cohort before the train/validation/test split. This may simply be a matter of presentation, but if that is correct, the normalization uses information from validation and test patients. Could the authors confirm whether this is the case, and if possible report whether recomputing the normalization statistics on the training split only materially changes the results?

5 comments
AuthorsOpenReview ↗

Response to Reviewer iuNm (1/3)

We thank the reviewer for reading the released code line by line. The finding about the reshape was correct, and it made the paper better. Both models are fixed and retrained, the two are compared under one shared protocol, and each further question is answered with a new experiment.

iuNm-1 Released code inconsistent with the paper. The signal-attention reshape does not group the K variables of one time step; the “fixed permutation” argument does not hold; exact reconstruction only shows faithfulness to the implemented network.

Response. We confirm the reviewer's reading and fixed it. Signal attention now groups the K variables of each time step, as the paper describes. Every affected model is retrained, and every number in the revision comes from the fixed network. After the fix, token t of the sequence entering retention is built only from the K variables at time step t, so the sequence is in clinical time order. The “fixed permutation” passage is removed from the paper.

Origin. The reshape comes from GARLIC's released code (Wang et al., 2026), from which TRACE inherits its signal-attention module. In the official repository, SignalAttentionLayer.forward in model.py permutes the tensor to [B, K, L, D] and reshapes it to [B·L, K, D] without exchanging the two axes, whereas Eq. 6 of the GARLIC paper defines the attention over the K signals of one time step. We applied the same fix to GARLIC and retrained it, so Table 1 compares two corrected networks.

Exactness. We re-verified exactness on the new checkpoints over the full test splits with five seeds. The mean relative gap is 1.0 × 10⁻⁵ on P12 and 1.2 × 10⁻⁶ on P19 in single precision, and below 10⁻¹³ in double precision (Table 2, Appendix A.2). The decomposition is exact for the network that the paper describes.

iuNm-2 Unequal tuning. TRACE tuned with Optuna on the split; GARLIC uses paper configs without retuning; checkpoint selection not matched; reproduced RETAIN much stronger than in the GARLIC paper. Retune GARLIC under the same split, budget and selection.

Response. TRACE and GARLIC are now compared under one protocol, and TRACE is ahead on all six cells of Table 1. On P12 and P19 the two models share the split, the search budget and the model-selection rule (Appendices D and F).

  • Split. One fixed stratified split, shared by every model and every seed.
  • Architecture. TRACE adopts GARLIC's published architecture hyperparameters and training schedule as they are and does not retune them.
  • Search budget. Both models are tuned on an 18-point grid over the same three axes, the classifier learning rate, its weight decay and the number of layers.
  • Selection. For both models the grid point with the highest validation AUPRC is trained with five seeds. Within a run both keep the checkpoint with the lowest validation classification loss, the rule of GARLIC's released training loop.

Beyond accuracy, TRACE's attribution is exact and unique where GARLIC's is not (iuNm-4).

RETAIN. The two papers treat baselines differently. The GARLIC paper keeps each baseline at the settings of its original publication wherever these exist (its Appendix C.1). We tune every baseline on our split. RETAIN is selected among 27 configurations of hidden size, embedding size and learning rate by validation AUPRC (Appendix F, Tables 6 and 7). On P12 the two reproductions agree (AUROC 83.5 against 83.1), and our tuned RETAIN is stronger on P19 (86.2 against 78.1) and on MIMIC-III (88.9 against 82.4). A stronger baseline makes the comparison harder for TRACE, not easier.

References

  • Wang et al. GARLIC: Graph attention-based relational learning of multivariate time series in intensive care. ICLR 2026.
AuthorsOpenReview ↗

Response to Reviewer iuNm (2/3)

iuNm-3 gROAR changes both value and mask. The retrained model can exploit the attribution-induced mask pattern; imputation can reconstruct removed values.

Response. gROAR removes a measurement completely, its value and its mask, so the imputation can draw only on the measurements a ranking kept. To verify that the gap between exact attribution and random selection does not come from the attribution-induced mask pattern, we ran a control on the same selected cells that removes only the values of the others and leaves the mask untouched, so every method sees the identical mask (Appendix G).

The gap does not come from the mask. Exact attribution beats random selection in 25 of 25 seed-and-budget pairs on each dataset. On P12 the gap widens at every budget once the mask is held fixed, from +12.3 to +20.0 AUPRC at 10% and from +2.4 to +6.8 at 50%. On P19 the top 10% of the values by exact attribution restore 58.1 AUPRC of the full model's 58.9.

P19, top 10% kept, test AUPRCexact attributionrandom selectionfull input
values removed, mask kept (control)58.152.558.9
values and mask removed (gROAR)45.918.058.9

Why gROAR removes the mask. With the mask left in place, a random 10% of the values already reaches 52.5 of 58.9. The mask carries the informative missingness (Che et al., 2018) of the original observations to every ranking, which leaves 6.4 points for any ranking to win, and exact attribution takes 5.6 of them. Under gROAR the same random selection falls to 18.0 and the rankings separate. Informative missingness is a property of the observation process, not of the attribution. Removing value and mask together is the design that tests the ranking itself. The perturbation tests without retraining, where no mask pattern can be learned, lead to the same conclusion (Appendix H).

References

  • Che et al. Recurrent neural networks for multivariate time series with missing values. Scientific Reports, 2018.
AuthorsOpenReview ↗

Response to Reviewer iuNm (3/3)

iuNm-4 Frozen-gate GRU. Could a frozen-gate convention be defined for a GRU? Would it fail to be exact, unique, or just be less convenient?

Response. It fails to be unique. This question led to a new experiment and a sharper Proposition 1. Following the reviewer's suggestion, we defined the frozen-gate decomposition for a GRU and ran it on GARLIC's own checkpoints (Appendix J, Table 15).

It is exact. With the gates z_t and r_t frozen and tanh(a_t) written as a slope times a_t, the GRU becomes a linear recursion h_t = A_t h_{t-1} + B_t x_t + c_t, and its unrolling sums to the logit at round-off.

It is not unique. The slope depends on the reference point chosen for tanh, and this choice enters at every step between an input and the output. We compared two standard rules, the zero-reference ratio tanh(a)/a of the rescale rule (Shrikumar et al., 2017) and the local slope tanh′(a). Both add up to the logit, and they disagree on where the credit goes.

Five seeds, whole test splitP12 GARLIC (GRU)P12 TRACEP19 GARLIC (GRU)P19 TRACE
Completeness gap, single precision (median)1.3 × 10⁻⁶1.4 × 10⁻⁶3.5 × 10⁻⁷2.8 × 10⁻⁷
Rank agreement of the two rules0.901.000.911.00
Top-10% overlap of the two rules (Jaccard)0.821.000.681.00
Rank agreement across training seeds0.720.820.780.87

On P19 two equally valid rules give GARLIC top-10% sets with a Jaccard index of 0.68, so a clinician would be shown a different explanation depending on a choice that the model does not determine. In retention the contribution of step j to step t is the single number R[t, j]. No pointwise nonlinearity lies between time steps, so no rule has to be chosen there. Retention is therefore what makes an exact attribution unique. Proposition 1 is rewritten to state exactly this.

iuNm-5 z-score statistics over the full cohort. Report train-only normalisation.

Response. Recomputing the statistics on the training split only does not change the results meaningfully, i.e., the difference is very marginal (Appendix M, Table 17). The split, the configuration and the seeds are the same.

TRACE, five seedsP12 AUROCP12 AUPRCP19 AUROCP19 AUPRC
whole-cohort statistics (Table 1)86.83 ± 0.9058.87 ± 1.9791.54 ± 0.4758.93 ± 2.64
training-split statistics86.68 ± 1.0258.24 ± 2.2091.56 ± 0.6358.13 ± 3.15

All four differences are at most 0.8 points and inside the seed standard deviation. Whole-cohort statistics are the convention of the released loaders of this benchmark. GARLIC (Wang et al., 2026) z-scores the whole cohort before splitting (load_data in utils.py), and Latent ODE (Rubanova et al., 2019), mTAND (Shukla & Marlin, 2021) and Warpformer (Zhang et al., 2023) compute their statistics on the full dataset as well. Our loader follows the same practice. The convention and the train-only result are now stated in the paper.

We hope these additions address the reviewer's concerns.

References

  • Wang et al. GARLIC: Graph attention-based relational learning of multivariate time series in intensive care. ICLR 2026.
  • Shrikumar, Greenside and Kundaje. Learning important features through propagating activation differences. ICML 2017.
  • Rubanova, Chen and Duvenaud. Latent ordinary differential equations for irregularly-sampled time series. NeurIPS 2019.
  • Shukla and Marlin. Multi-time attention networks for irregularly sampled time series. ICLR 2021.
  • Zhang et al. Warpformer: A multi-scale modeling approach for irregular clinical time series. KDD 2023.
Reviewer_s8Q6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TRACE, an interpretable framework for irregular clinical time-series prediction that aims to achieve accurate predictions and efficient, faithful explanations. The method builds upon a graph-attention architecture and replaces the GRU temporal encoder with softmax-free retention, enabling single-pass, signed attribution with an exact decomposition of prediction logits under fixed attention weights. The paper also introduces graded ROAR (gROAR) to evaluate attribution quality across different feature-retention budgets. Experiments on three ICU benchmarks show that TRACE achieves competitive or superior predictive performance against nine baselines. Additional evaluations demonstrate near-machine-precision completeness, substantially faster attribution than Integrated Gradients, and comparable attribution quality under gROAR.

Critical strengths
  1. The architecture is well designed, preserving the ability to model irregular sampling and cross-variable dependencies while enabling fine-grained, signed attribution. More importantly, it achieves this without sacrificing predictive performance, making interpretability a practical part of the model rather than an expensive post-hoc addition.

  2. The paper carefully validates its theoretical claims through independent bias accumulation and doubleprecision experiments, providing convincing evidence for numerical exactness rather than relying solely on theoretical derivations.

  3. The attribution evaluation is well controlled, particularly the same-model comparisons that isolate attribution quality from model capacity. The results demonstrate that TRACE achieves attribution quality comparable to Integrated Gradients at a substantially lower computational cost.

Critical weaknesses
  1. TRACE freezes attention weights and only attributes the value path. While the decomposition is numerically exact, it does not fully explain the model's decision-making process, raising questions about the claimed attribution faithfulness.

  2. The necessity of the Retention architecture is not sufficiently justified. More controlled comparisons with alternative temporal encoders would help establish whether Retention offers unique advantages.

  3. Although gROAR demonstrates that selected features are predictive after retraining, it does not directly measure their influence on the original model's predictions. Additional perturbation-based evaluations would strengthen the faithfulness claims.

Questions for authors
  1. Given that attention weights are frozen during attribution, to what extent does the proposed decomposition capture the model's actual decision-making process beyond numerical completeness?

  2. Could the authors provide more controlled comparisons with alternative temporal encoders to clarify whether softmax-free Retention is essential for achieving the claimed attribution properties?

  3. Could the authors evaluate attribution faithfulness through direct perturbations of the original model, rather than relying primarily on retraining-based gROAR evaluations?

2 comments
AuthorsOpenReview ↗

Response to Reviewer s8Q6 (1/2)

We thank the reviewer for the careful assessment. We have conducted the additional experiments requested in all three questions and incorporated the results into the revised manuscript.

s8Q6-1 Attention weights are frozen and only the value path is attributed. Does the decomposition capture the decision process beyond numerical completeness?

Response. It does. Holding the attention weights at their evaluated values is the standard convention for attributing attention models (Ali et al., 2022). Whether the resulting attribution captures the decision is an empirical question. We test it on the same TRACE checkpoint in two ways, without retraining and with retraining. In both, the reference is Integrated Gradients (IG; Sundararajan et al., 2017).

Without retraining (Figure 8, Table 13). We progressively remove the highest-ranked cells from the input of the trained model and measure the resulting decline in its prediction score. A faithful attribution ranking should produce a rapid decline, indicating that the most influential cells have been correctly identified.

Area for k ≤ 50%P12 Deletion↓P12 Compreh.↑P19 Deletion↓P19 Compreh.↑
TRACE exact (1 pass)37.22.2335.20.82
IG (50 passes)40.52.1136.80.42
attention weights41.70.4940.1−0.32
random57.10.1853.0−0.10

Exact attribution degrades the prediction most on both datasets. Ranking by the attention weights, the quantity we hold fixed, changes it far less.

With retraining (Figure 2, bottom row, and Table 9). gROAR keeps only the top-k% cells of a ranking and retrains the model on them. A ranking that captures the decision keeps the accuracy high with few cells.

Test AUPRC, top 10% keptP12P19
TRACE exact53.645.9
IG52.144.7
attention weights40.132.8
random41.318.0
full input58.958.9

Exact attribution beats random selection at every budget from 10% to 50%. With 10% of the cells it keeps most of the full model's accuracy, while the attention ranking falls to the level of random selection on P12.

Both tests agree. The cells that exact attribution ranks highest are the ones the model relies on, so holding the attention weights fixed does not hide the decision process. IG differentiates through the attention weights and freezes nothing, and exact attribution matches it in a single backward pass (Table 2).

References

  • Sundararajan, Taly and Yan. Axiomatic attribution for deep networks. ICML 2017.
  • Ali et al. XAI for Transformers: Better explanations through conservative propagation. ICML 2022.
AuthorsOpenReview ↗

Response to Reviewer s8Q6 (2/2)

s8Q6-2 Necessity of retention not justified. Controlled comparisons with alternative temporal encoders.

Response. Retention (Sun et al., 2023) is essential for the attribution property that matters, which is uniqueness. We compare it with two alternatives, the model without retention and the model with a GRU in its place.

Without retention. We remove the retention stack in an ablation study (Table 16), and both AUROC and AUPRC drop on P12 and P19.

With a GRU. GARLIC shares TRACE's front end and uses a GRU core, which makes it the controlled comparison. Both models use the same code, the same architecture hyperparameters, a tuning grid of the same size and the same seeds.

P12 GARLIC (GRU)P12 TRACEP19 GARLIC (GRU)P19 TRACE
Test AUROC / AUPRC84.9 / 53.886.8 / 58.991.3 / 57.791.5 / 58.9
Exact decomposition existsyesyesyesyes
Top-10% overlap of two rules (Jaccard)0.821.000.681.00
Rank agreement across training seeds0.720.820.780.87

Retention is not only more accurate than a GRU, it also gives an attribution that is unique and more stable across training seeds. A GRU with frozen gates can be decomposed exactly, but the answer depends on how tanh is linearized, and two standard rules disagree on the most important cells. Retention has no nonlinearity between time steps, so its ranking is identical under both rules (Proposition 1, Appendix J).

s8Q6-3 gROAR measures predictiveness after retraining, not influence on the original model.

Response. We added direct perturbations of the original model, with no retraining (Appendix H, Figure 8, Table 13). The areas are in the table under s8Q6-1. Exact attribution has the strongest influence on the original model's prediction on both datasets, with the lowest deletion area and the highest comprehensiveness. The perturbation tests and gROAR agree. The cells that TRACE ranks highest are the ones the trained model relies on, and they are also the ones a retrained model needs.

We hope these additions address the reviewer's concerns.

References

  • Sun et al. Retentive network: A successor to Transformer for large language models. arXiv:2307.08621, 2023.
Reviewer_hEZPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces an interpretable model called TRACE for irregularly sampled ICU time series. It keeps the front end of GARLIC and replaces the GRU with softmax-free retention, so that the prediction logit can be decomposed into signed per-variable, per-timestep contributions in a single backward pass. Experiments show better performance compared to the baselines.

Critical strengths

TRACE gets the highest AUROC and AUPRC on three ICU benchmarks (Table 1), demonstrating that interpretability does not have to reduce predictive performance. Also, the attribution is computed in a single backward pass, with a speedup over IG (Table 2).

Critical weaknesses
  • The paper’s novelty is quite weak. First, the TRACE architecture is built based on GARLIC [1] by replacing the GRU with softmax-free retention (Sec. 3.2). Second, the explanation approach is a gradient-based method (Eq. 2, the DeepLIFT rescale rule), and decomposing the logit into input-gradient contributions plus a bias term (Eq. 1) has been studied before [2, 4]. The paper does not discuss [4].
  • The TRACE explanation approach is only evaluated when the task is binary classification, which the authors acknowledge as a limitation (Sec. 5). Please discuss if the proposed method can be used for multi-label classification, which other explanation approaches, such as post-hoc explanations [3], can do.
  • The paper lacks an ablation study of the TRACE architecture to study the impact of each stage on the overall performance. The effect of each loss term on the overall performance is also missing.
  • The evaluation of faithfulness in Table 2 relies only on the relative completeness gap. Besides gROAR (Tables 3-4), which requires retraining, evaluating the explanation on other aspects, such as deletion and insertion tests without retraining, comprehensiveness and sufficiency, and infidelity, would make the claim stronger.
  • TRACE has many hyperparameters. In Table 6, I see that 12 hyperparameters are tuned, while several others are fixed but the authors do not provide justification, such as the reconstruction weight and the retention decay γ. Plus, there is no hyperparameter sensitivity analysis provided. It is therefore unclear whether the performance and the attributions stay stable under different hyperparameters.

[1] Li, Yanke, et al. “GARLIC: Graph Attention-based Relational Learning of Multivariate Time Series in Intensive Care.” The Fourteenth International Conference on Learning Representations. 2026.

[2] Wang, Yongjie, et al. “Gradient based feature attribution in explainable AI: A technical review.” arXiv preprint arXiv:2403.10415 (2024).

[3] Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. “Why should I trust you?’ Explaining the predictions of any classifier.” Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2016.

[4] Srinivas, Suraj, and François Fleuret. “Full-gradient representation for neural network visualization.” Advances in neural information processing systems 32 (2019).

Questions for authors

Please address the above weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer hEZP (1/3)

We thank the reviewer for providing this specific and actionable list of suggestions.

hEZP-1 Novelty. The architecture is GARLIC with the GRU replaced by softmax-free retention; the explanation is the DeepLIFT rescale rule; logit = input-gradient contributions + bias is already studied (gradient-attribution review arXiv:2403.10415; FullGrad, not discussed).

Response. Our novelty is threefold. Each part closes a gap that prior work leaves open.

  1. First exact attribution at top accuracy. An attribution is exact when the credits of all measurements, plus a bias, add up to the model's logit. Before TRACE, an interpretable model for irregular clinical time series offered either this or strong accuracy, not both. RETAIN (Choi et al., 2016) is exact because its prediction stays linear in the input embeddings, and TRACE's AUPRC is 9% to 30% higher on the three benchmarks. IMV-LSTM (Guo et al., 2019) and GARLIC (Wang et al., 2026) explain through attention weights, which do not add up to the prediction and need not be faithful (Jain & Wallace, 2019). TRACE gives the exact decomposition in one backward pass (Table 2). It also has the best AUROC and AUPRC among interpretable models and a higher AUPRC than all nine baselines (Table 1).
  2. An architecture built for exactness. FullGrad (Srinivas & Fleuret, 2019) proves completeness for ReLU networks, which do not incorporate attention mechanisms or gating operations. DeepLIFT (Shrikumar et al., 2017) explains a difference from a reference input that has to be chosen. Integrated Gradients (Sundararajan et al., 2017) needs a baseline and many passes, and on our checkpoints it still misses its target by 6% and 2% after 50 passes. Gradient×Input has no completeness guarantee and misses by a factor of 3.2 and 6.3 (Table 2). These methods are well established, but each holds only under its own conditions. If we decompose a GRU exactly with its gates frozen, the answer depends on how tanh is linearized, and two standard rules disagree on the most important cells (Appendix J, Table 15). Applying these rules directly to an existing model therefore does not guarantee an exact and unique attribution. TRACE is explicitly designed to guarantee exact attribution through its architecture. Its temporal core, softmax-free retention, eliminates nonlinear transformations between time steps. The effect of a past measurement on the prediction is then a single number, with no reference, baseline or linearization rule to choose along time, and one backward pass gives every measurement its exact share of the logit (Proposition 1). We now cite and discuss the gradient-attribution review (Wang et al., 2024) and FullGrad (Srinivas & Fleuret, 2019) in Section 2 and Appendix I (Table 14).
  3. A validated faithfulness test. Prior work (Wang et al., 2026) tests an explanation at a single 50% budget, which fails to reliably distinguish attribution rankings. Keeping the top 50% cannot tell rankings apart, and removing the top 50% wipes out whole variables rather than identifying the important ones. With the attention ranking, 22 of the 36 variables of P12 keep less than 1% of their observations. Graded ROAR (gROAR) instead compares every ranking with a random mask of the same size at each budget, building on ROAR (Hooker et al., 2019). We validate it with a control that keeps the observation mask and with perturbation tests without retraining (Appendices G and H). TRACE is the only interpretable model whose ranking beats a random mask at every budget (Table 12).

References

  • Wang et al. GARLIC: Graph attention-based relational learning of multivariate time series in intensive care. ICLR 2026.
  • Wang et al. Gradient based feature attribution in explainable AI: A technical review. arXiv:2403.10415, 2024.
  • Srinivas and Fleuret. Full-gradient representation for neural network visualization. NeurIPS 2019.
  • Choi et al. RETAIN: An interpretable predictive model for healthcare using reverse time attention mechanism. NeurIPS 2016.
  • Guo, Lin and Antulov-Fantulin. Exploring interpretable LSTM neural networks over multi-variable data. ICML 2019.
  • Jain and Wallace. Attention is not explanation. NAACL 2019.
  • Sundararajan, Taly and Yan. Axiomatic attribution for deep networks. ICML 2017.
  • Shrikumar, Greenside and Kundaje. Learning important features through propagating activation differences. ICML 2017.
  • Hooker et al. A benchmark for interpretability methods in deep neural networks. NeurIPS 2019.
AuthorsOpenReview ↗

Response to Reviewer hEZP (2/3)

hEZP-2 Only binary classification. Discuss multi-label; post-hoc methods such as LIME can handle it.

Response. TRACE extends naturally to multi-class and multi-label outputs without any architectural modifications. The decomposition works on one logit at a time and does not depend on the number of outputs. With M outputs, one forward pass and one backward pass per output give M exact decompositions, z^m=∑k,tCk,t(m)+b(m)\hat z_m=\sum_{k,t}C^{(m)}_{k,t}+b^{(m)}, each adding up to its own logit. The margin between two classes is decomposed exactly by the difference of their two maps. LIME (Ribeiro et al., 2016) also handles several outputs, but it fits an approximate local model for each one from many perturbed inputs. TRACE stays exact for every output. Appendix I now states this.

hEZP-3 No ablation of each stage or loss term.

Response. We added the ablation of every stage and of both loss terms (Appendix M, Table 16). Each row removes one component, retrains the model with five seeds under the protocol of Table 1, and reports the change against the full model.

Removed componentP12 ΔAUROCP12 ΔAUPRCP19 ΔAUROCP19 ΔAUPRC
Retention core
temporal module (LayerNorm and pooling only)−0.92−2.52−0.78−2.45
FFN−0.37−2.39−0.49−2.85
learned decay (γ = 1)−0.45−1.01−0.12−1.87
Other stages
time-lag attention−0.13+0.04−2.09−7.61
message passing−1.46−4.46−0.99−3.21
signal attention−0.27−0.60−0.40−0.96
missing indicator−0.92−2.25−9.32−25.78
decay imputation (mean fill)−2.03−4.02−0.79−1.27
per-variable encoders (one shared encoder)−0.54−0.98−1.18−3.79
Auxiliary losses
λ_rec = 0+0.20+0.15−0.32−1.92
λ_graph = 0−0.14−1.14−0.36−1.94

Without the retention module, AUPRC falls by about 2.5 points on both datasets. Among the other stages, message passing and decay imputation matter most on P12, and the missing indicator and time-lag attention matter most on P19, where removing them lowers AUPRC by 44% and 13%.

References

  • Ribeiro, Singh and Guestrin. “Why should I trust you?” Explaining the predictions of any classifier. KDD 2016.
AuthorsOpenReview ↗

Response to Reviewer hEZP (3/3)

hEZP-4 Faithfulness only via completeness gap and gROAR. Add deletion/insertion without retraining, comprehensiveness/sufficiency, infidelity.

Response. We have incorporated all of these evaluation metrics using the trained checkpoints from Table 1, without retraining (Appendix H, Figure 8, Table 13). Each test removes, or keeps only, the top-k% cells of a ranking and reads the output of the trained model. Deletion and insertion (Petsiuk et al., 2018) evaluate changes in test AUPRC, while comprehensiveness and sufficiency (DeYoung et al., 2020) quantify changes in the predicted-class logit. A faithful ranking makes the deletion curve fall quickly and the comprehensiveness curve rise quickly.

Area for k ≤ 50%P12 Del.↓P12 Ins.↑P12 Comp.↑P12 Suff.↓P19 Del.↓P19 Ins.↑P19 Comp.↑P19 Suff.↓
TRACE exact (1 pass)37.256.02.23−1.8235.244.80.82−0.37
IG (50 passes)40.556.52.11−1.7436.844.30.42−0.41
Gradient×Input41.956.42.02−1.6537.644.10.27−0.16
attention weights41.746.20.490.5040.139.2−0.320.57
random57.147.60.180.6653.028.6−0.100.07

Exact attribution has the lowest deletion area and the highest comprehensiveness on both datasets. On insertion and sufficiency it is level with IG, with one pass instead of 50. On infidelity (Yeh et al., 2019) it is level with IG and better than Gradient×Input. These tests and gROAR lead to the same conclusion.

hEZP-5 Twelve tuned hyperparameters; λ_rec and γ fixed without justification; no sensitivity analysis.

Response. TRACE now tunes only three hyperparameters on P12 and P19, the classifier learning rate, its weight decay and the number of retention layers, over an 18-point grid (Appendix D, Table 4). We fix the architecture, the training schedule and the reconstruction and graph weights, all taken from GARLIC's published configuration, so λ_rec is fixed. The decay γ is now learned per head and ablated in Table 16, and we see a drop on both datasets when it is fixed at γ = 1.

A sensitivity analysis is now added in Appendix D (Figure 4). Each bar is the change of the mean test AUPRC when one setting is varied. Seventeen of the eighteen bars are shorter than the seed-to-seed standard deviation of the final model. The exception is the hidden size on P12, which TRACE does not tune and takes from GARLIC. Our attribution is also more stable across training seeds than GARLIC's (Table 15).

We hope these additions address the reviewer's concerns.

References

  • Petsiuk, Das and Saenko. RISE: Randomized input sampling for explanation of black-box models. BMVC 2018.
  • DeYoung et al. ERASER: A benchmark to evaluate rationalized NLP models. ACL 2020.
  • Yeh et al. On the (in)fidelity and sensitivity of explanations. NeurIPS 2019.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading, and Reviewer iuNm in particular for inspecting the released code. The reviews made the paper stronger. Every model is retrained on corrected code under one fair protocol, and each of our three contributions now rests on new evidence. All changes in the revised PDF are marked in blue.

  1. Code fix (iuNm-1). We confirm the reviewer's reading of the signal-attention reshape. The same reshape is present in GARLIC's released code, from which TRACE inherits its signal-attention module. We fixed the grouping in both models, then retrained every affected model in the paper, retuning on P12 and P19 under the protocol of point 2. Exactness is re-verified on the new checkpoints.
  2. Fair tuning protocol (iuNm-2, hEZP-5). On P12 and P19, TRACE now keeps GARLIC's published architecture and training schedule and tunes only three settings, the classifier learning rate, its weight decay and the number of retention layers, on an 18-point grid. We further retuned GARLIC, with the fix of point 1, on a grid of the same size and axes. MIMIC-III keeps the submitted configuration for both models, retrained with the fix. Both models are selected by validation AUPRC, and every downstream table is recomputed under this protocol.
  3. Faithfulness without retraining (hEZP-4, s8Q6-3). On the checkpoints above we ran perturbation tests with no retraining. Deletion and insertion curves and comprehensiveness and sufficiency curves are reported on P12 and P19 with five seeds each.
  4. Architecture ablations (hEZP-3, s8Q6-2). We conduct architecture ablations with five seeds per arm on P12 and P19. First, for the retention core, we remove the FFN, the learned decay, and the temporal module itself. Second, we remove the other stages of the architecture: time-lag attention, message passing, signal attention, the missing indicator, decay imputation and the per-variable encoders. Third, we ablate the two auxiliary losses, λ_rec and λ_graph.
  5. gROAR design (iuNm-3). gROAR removes a measurement completely, its value and its mask. To verify the gap between exact attribution and random selection, we ran a control that removes only the value. The gap does not come from the mask. Exact attribution still beats random selection in 25 of 25 seed-and-budget pairs on each dataset. The control also shows that a mask left in place carries the informative missingness of the original observations to every ranking. Removing both the value and the mask is therefore the gROAR design that tests the ranking itself.
  6. Why retention (iuNm-4, s8Q6-2). Following iuNm-4, we ran the frozen-gate decomposition on GARLIC's own GRU checkpoints. It is exact, but the attribution varies with the rule used to linearize tanh. We examine two standard rules, and on P12 and P19 their top-10% sets overlap with a Jaccard index of only 0.82 and 0.68, where 1 means the same set. Retention's ranking is identical under both. Proposition 1 now states this.

The remaining points are answered individually below. In total the revision rests on about 700 training runs. We appreciate the reviewers' time and effort. Every point raised is addressed, and we are glad to clarify anything further during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.