- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes GapTrace, a framework for analyzing prediction disparities in clinical time-series models. It combines fairness measurement, feature attribution, statistical testing, and controlled interventions to identify features associated with prediction gaps across patient groups.
Critical strengths
The paper addresses an important problem in trustworthy machine learning. The motivation is clear, and the experimental evaluation is relatively comprehensive. I particularly appreciate the effort to connect fairness measurement, differential attribution, and intervention-based validation into a unified framework.
Critical weaknesses
-
Novelty. The framework primarily combines existing techniques, including Integrated Gradients, DeepLIFT, statistical testing, feature clustering, and distributional interventions. While the integration is useful, I am not fully convinced that it constitutes a sufficiently novel methodological contribution. I would encourage the authors to further clarify what GapTrace offers beyond a straightforward combination of existing attribution and fairness auditing methods, ideally through additional comparisons or ablation studies.
-
Explanatory power. In the MIMIC-IV experiments, the prediction gap between age groups is 29.7 percentage points, while the proposed intervention reduces it by only approximately 2.3 percentage points. This raises a question about how much of the observed disparity the framework can actually explain. Could the authors further investigate whether the remaining gap reflects limitations of the attribution methods or differences in model behavior that cannot be captured by the selected features?
-
Intervention validity. The proposed interventions may disrupt dependencies among clinical features and generate unrealistic inputs. I encourage the authors to further discuss how this limitation affects the interpretation of the intervention results and whether more distribution-preserving interventions could strengthen the conclusions.
Questions for authors
-
Novelty: What are the key methodological contributions of GapTrace beyond combining existing attribution, statistical testing, and fairness auditing techniques? Could the authors provide additional comparisons or ablation studies to better demonstrate its unique advantages?
-
Explanatory Power: In the MIMIC-IV experiments, the intervention reduces the age-related prediction gap by only 2.3 out of 29.7 percentage points. How should we interpret the relatively limited reduction? Does it reflect limitations of the attribution methods, the selected features, or the intervention procedure itself?
-
Intervention Validity: Since the proposed interventions may disrupt dependencies among clinical features, how can we ensure that the resulting gap reductions reflect meaningful model behavior rather than artifacts of unrealistic inputs? Could more distribution-preserving interventions help validate the findings?
2 comments
- Intervention validity: We acknowledge that modifying clinical features may disrupt physiological and temporal dependencies, potentially producing out-of-distribution inputs. GapTrace's interventions are designed as diagnostic probes of a frozen model, rather than causal counterfactuals or clinically realizable interventions. To improve their interpretability, we employ three complementary strategies:
- Rank-preserving distributional repair: Aligns group-wise feature distributions while preserving within-group ranks, although multivariate dependencies may change.
- Joint feature transplantation (I2, Appendix A.2.7): Transfers selected features jointly from observed donor samples, preserving dependencies within the transplanted subset.
- Matched-control interventions: Compare targeted modifications against the same perturbation applied to other model-relevant features selected for comparable attribution magnitude.
These strategies help assess feature-specific sensitivity, although none guarantees that modified inputs remain on the clinical data manifold. Accordingly, intervention results should be interpreted as model sensitivity under specified perturbations, not causal effects. The observed 2.3-percentage-point reduction exceeds matched random repairs (p = 0.048), but does not eliminate concerns about intervention realism.
We have strengthened the revised manuscript with an outcome-conditional performance-gap analysis (Appendix A.9.2), separating the contributions of static diagnosis codes and time-series inputs. We have also clarified that GapTrace's interventions are diagnostic probes of model sensitivity rather than causal counterfactuals. More clinically constrained interventions, including conditional sampling and matched trajectory transplantation, remain important directions for future work.
We thank the reviewer for the careful and constructive assessment.
In particular, GapTrace connects statistically supported differences in model attribution, correlated and temporally structured feature groups, and targeted interventions evaluated against matched controls. These stages address distinct questions: whether model reliance differs across groups, which features exhibit these differences, and whether modifying those features changes the prediction gap. We have clarified these contributions and their relationship to existing fairness-attribution methods in the revised Introduction and Related Work sections. The revised evaluation also includes a FairCanary-inspired baseline (Table 1), against which GapTrace achieves higher feature-recovery F1 (0.92 vs. 0.79) and balanced accuracy (0.94 vs. 0.84) on the controlled synthetic benchmark.
Several factors may contribute to the remaining disparity. The intervention modifies only selected features, while the model may retain age-associated information through other variables, correlated measurements, or interactions. Furthermore, marginal distributional alignment does not necessarily remove differences in joint dependencies or temporal relationships. Attribution-based selection is also sensitive to the attribution method and reference distribution.
To further investigate the scope of the intervention, we added an outcome-conditional performance-gap decomposition (Appendix A.9.2, Tables 13--14). Among survivors, patients aged ≥65 exhibit a significant binary-cross-entropy error gap of 0.721 (p=0.001) relative to younger patients. Approximately 95% of this gap is attributed to static diagnosis codes (0.683), compared with 5% to time-series inputs (0.038). Five diagnosis codes show significant attribution differences after FDR correction, with the five largest contributions accounting for approximately 65% of the error gap. Since the evaluated intervention modifies selected time-series features rather than static diagnosis codes, these findings provide additional context for its measured effect. Importantly, this decomposition concerns outcome-conditional prediction error, not the original 29.7-percentage-point prediction gap.
The extended analysis provides a more comprehensive characterization of age-associated disparities by examining contributions from both static diagnosis codes and time-series features. Together, these complementary analyses provide a broader understanding of the observed disparities.