acceptodds
Under review as a conference paper at ICLR 2027

A Faithful Attribution Reports Its Head, Not Its Model: The Limits of Completeness in Additively Decomposed Representations

Abstract

Additive models promise an explanation readable off the architecture: the latent is a sum of per-component terms, so each contribution is available without a post-hoc estimator. That promise is bounded by the downstream head. A completeness-satisfying attribution decomposes a head's output, so what it can say is fixed by the head you chose, not by the model. On MD-TJEPA, a JEPA whose latent decomposes over channel band scale, channel identity is decodable from the components and only (chance ) from the sum they form. Fit the head for the channel and its exact decomposition closes of the gap to a pre-aggregation score (, ); on a plain NAM the same construction is not weak but arbitrary, or in most seeds. Fit it for the model's own task — a band detector whose labels contain no channel — and an equally exact decomposition names the channel at . An input intervention separates the two: resampling the injected channel moves both heads' outputs, moving its content to another channel barely does, so the sum hides which channel contributed, not whether it did. The band head's map agrees with that intervention on of windows, the channel head's on ; most of each map's disagreement is its implicit reference, the latent origin. Re-referenced to the training-mean contribution — on this synthetic benchmark an estimate of the expected intervention — they agree on and . Completeness pins the sum, not the head and not the reference. We are scoped on faithfulness, not accuracy — TS2Vec outperforms MD-TJEPA under a matched probe, though an untrained MD-TJEPA read through its components outperforms TS2Vec — and the axes conflated as "interpretability" disagree on the same checkpoints. What this buys is a check worth running first: probe the aggregate for the axis, and intervene on the input for it, before trusting any attribution over it — a probe near chance says the aggregate hides which channel was active, not that its content goes unused. between and costs one line and exposed a gate collapse here — the JEPA objective shares the gate between prediction and target, so an unopposed sparsity penalty switches components off. In a minimal additive stand-in, removing the gate from the target prevents that shrinkage outright. We close with three checks to run before reading any additive attribution.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, attribution, additive models, self-supervised learning, joint-embedding predictive architecture, time series, biosignals, faithfulness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uDPnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the limitations of completeness in additive attribution models. It shows that attribution rankings depend on the choice of downstream head and reference, even when completeness is satisfied. The authors use an additive time-series model and a Neural Additive Model to examine these limitations through theoretical analysis and empirical evaluation.

Critical strengths
  1. The paper includes extensive experiments across synthetic and real-world datasets. These experiments provide useful observations about the behavior of additive attribution.
Critical weaknesses
  1. The main observation that completeness is insufficient to guarantee faithful attribution has already been extensively studied, including even theoretical results such as Bilodeau et al. (PNAS’2024). The paper mainly examines this known limitation in a specific additive setting, without establishing a sufficiently novel theoretical insight.

  2. The core theoretical analysis is restricted to additive latent representations with linear downstream heads. This is a relatively narrow setting compared with general attribution methods, which can explain nonlinear models at dffierent modules or input level. It is unclear how the findings extend to broader attribution settings.

  3. The paper covers many loosely connected findings, such as attribution limitations, downstream performance, evaluation, etc. The extensive discussion and secondary observations makes the main contribution difficult to follow. A more focused presentation would improve clarity.

Questions for authors
  1. Given that the insufficiency of completeness is already well established, what fundamentally new theoretical insight does this paper provide beyond existing works?

  2. What generalizable approaches for improving attribution reliability can be derived from these findings, particularly for attribution methods beyond additive representations with linear heads?

Reviewer_ZXKmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies intrinsic attribution in models whose latent representation is an additive sum of per-component terms read by a linear head. The authors show that, under completeness, local-linearity and a non-degeneracy assumption, the attribution is unique for a given head, and argue that what it reports therefore depends on the choice of head. They support this with experiments on MD-TJEPA, a time-series JEPA decomposed over channel, band and scale, mainly on a synthetic benchmark, together with an input intervention and additional analyses on real data.

Critical strengths

S1. The question itself matters for attribution research. Whether an exact decomposition that comes from the architecture can be taken as an explanation of the model is worth asking.

S2. The resample vs. swap intervention is useful. It separates "the axis cannot be recovered from the summed latent" from "the axis does not contribute to the output" in a clean way.

Critical weaknesses

W1. It is unclear whether the contribution goes much beyond what is already known. Proposition 1 follows almost directly from the linearity of the head. The setup also looks very close to CAM (Zhou et al., 2016): spatial positions play the role of components, and global average pooling plays the role of the additive aggregate. Seen this way, a head trained without the contested axis still localizing it is essentially what CAM's weakly supervised localization showed. CAM and Grad-CAM do not appear in the references. Compared with Bilodeau et al. (2024) and Sundararajan and Najmi (2020), it is not clear what is new apart from the setting.

W2. Most of the evidence comes from a synthetic benchmark where only one (channel, band) cell is active per sample, and on real data only a weaker version of the effect is reported (Appendix N).

W3. With block aggregation, the signed attribution tracks the probe on TempCNN but not on PatchTST (Section 4, Appendix D). The authors give a partial explanation and leave the rest open.

W4. The paper is hard to follow, mostly because the manuscript does not read as finished.

There are many "an earlier version said" or "we withdraw" statements, which makes it difficult to identify the final claims. Parts of the appendix seem to come from an earlier framing (e.g., Appendix A.3), and some internal notes are still there (e.g., "Phase 2 W30" in Appendix A.5.6). Section 3.2 and Appendix C also seem to disagree on what Proposition 1 needs.


[1] Zhou, Bolei, et al. "Learning deep features for discriminative localization." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

[2] Selvaraju, Ramprasaath R., et al. "Grad-cam: Visual explanations from deep networks via gradient-based localization." Proceedings of the IEEE international conference on computer vision. 2017.

[3] Bilodeau, Blair, et al. "Impossibility theorems for feature attribution." Proceedings of the National Academy of Sciences 121.2 (2024).

[4] Sundararajan, Mukund, and Amir Najmi. "The many Shapley values for model explanation." International conference on machine learning. PMLR, 2020.

Questions for authors

Q1. Would the dependence on the head still hold on real data with other additive models, beyond the synthetic setting?

Q2. For a linear head, the proposed attribution looks equivalent to IG with a zero baseline, and re-referencing to changing that baseline. For nonlinear heads, it is already known that IG or SHAP results depend on the head and the baseline. What do the conclusions add beyond these known facts?

Reviewer_wU33OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper examines the limits of completeness for intrinsic attributions in additively decomposed representations. For a sum of gated components read by a linear prediction head, the authors characterize the unique signed decomposition of the head’s score under locality, linearity, and nondegeneracy assumptions. Their central argument is that completeness constrains the sum of contributions but does not determine the head or attribution reference, and does not by itself establish signal-source localization or agreement with input interventions.

The study uses MD-TJEPA, a self-supervised time-series model with channel–frequency–scale components, as an experimental vehicle. It compares complete signed attributions with component-norm and magnitude readouts, varies prediction heads on a frozen encoder, and examines alternative aggregation structures and attribution references. A separate component-identifiability analysis requires a centering condition that the authors report does not hold in their trained models. The authors also diagnose gate collapse under shared prediction-target gates and sparsity penalties, and report that a retraining recipe incorporating a balance term keeps all components active.

Validation combines controlled synthetic localization tasks, a constructed neural additive-model example, and experiments on Sleep-EDFx and PTB-XL across multiple backbones and training seeds. Different heads on the same representation produce substantially different yet complete attributions; a task head can localize the contributing channel even when channel identity is poorly decoded from the aggregate. Preserving channels during aggregation restores channel decodability on both tested backbones, but attribution localization improves unevenly, and the downstream gain on Sleep-EDFx is small. Adjusting the attribution reference substantially improves agreement with channel-replacement interventions on the synthetic benchmark. Further audits show that ground-truth localization, parameter-randomization tests, downstream utility, and clinical alignment can disagree on the same checkpoints. MD-TJEPA also underperforms TS2Vec and its own untrained initialization under matched downstream probing on Sleep-EDFx. The contribution is framed as a theoretical and empirical account of these distinctions, together with practical checks for evaluating additive attributions beyond completeness.

Critical strengths

The paper separates source identification, decodability from the aggregate, and the effect of an input on a particular head. The frozen-encoder head comparisons and reference-adjusted intervention experiments make these distinctions concrete. The gate-collapse audit also provides a practical diagnostic for checking whether a displayed heatmap reflects the components actually used in computation. These are useful lessons, although the strongest evidence comes from controlled synthetic settings and the conditional theory provides limited insight into the behavior of the trained models.

Critical weaknesses

The main text distinguishes completeness, parameter sensitivity, source localization, and intervention agreement, but several appendices retain stronger interpretations that do not consistently respect these distinctions. Appendix A.3 describes MD-TJEPA as providing a by-construction, model-faithful attribution and frames the contribution as an interpretability-performance trade-off. Appendix A.6 repeatedly uses Pareto terminology, whereas Figure 2 explicitly states that the comparison combines different backbones and datasets and establishes an observed anti-correlation, not a Pareto frontier. Appendix A.5.7 treats sensitivity to encoder randomization as establishing faithfulness to the trained model, while Appendix P invokes a PTB-XL parameter-randomization pass in a discussion of perturbation-based faithfulness without a corresponding perturbation evaluation. These passages blur the validation criteria that the paper’s central argument asks readers to distinguish. Unreconciled reporting compounds the problem. Appendix A.6 describes the same gated-variant improvement as both 0.346->0.803 and 0.31->0.82 without identifying different settings, and several cross-references direct readers to unrelated experiments.

For the signed attribution considered here, dependence on the head and reference follows directly from the construction: changing the head changes the predictive function being explained, while changing the reference changes the comparison underlying the attribution. Reference dependence under axiomatic guarantees is already discussed by Sundararajan and Najmi (2020), and Covert et al. (2021) explicitly distinguish the behavior being explained from the removal convention. Bilodeau et al. (2024) establish a related, although technically different, limitation of completeness for counterfactual inference. The paper acknowledges these precedents, but the additional conceptual insight from specializing them to additive latent representations remains limited. Proposition 1 establishes uniqueness under local linearity and a strong nondegeneracy assumption; it does not characterize when a fixed head’s attribution will localize a source or agree with interventions. The separate identifiability theorem does not establish identifiability of the trained components because its centering condition is unmet. The contribution therefore rests primarily on the empirical audit. The strongest head and intervention experiments use exactly one injected oscillation against otherwise noise, while the tabular NAM deliberately ties component functions to make the aggregate group-blind. These are useful controlled illustrations, but their practical reach is uncertain. In the synthetic intervention experiment, the appropriately referenced score estimates the expected replacement effect by construction, limiting the independence of the agreement result. On real data, the gate-repaired models demonstrate ranking differences across related heads, but the reference-adjustment result lacks corresponding intervention validation. Thus, the paper provides useful clarification of attribution choices within its audited settings, but does not yet establish sufficiently substantial additional insight or practical consequences beyond the acknowledged principles.

Questions for authors
  1. What is the principal advance of Proposition 1, Corollary 1, and the empirical audit beyond the target and reference dependence discussed by Sundararajan and Najmi (2020), Covert et al. (2021), and the related limitations studied by Bilodeau et al. (2024)? 2.Which positive claims about faithfulness and the interpretability-performance trade-off in Appendices(e.g., A.3, A.4, A.5.7, A.6, P) do the authors retain? Please distinguish exact signed decomposition, parameter sensitivity, input dependence, and intervention agreement, identifying the readout and checkpoint supporting each retained claim. 3.Re-referencing to the training-mean contribution substantially improves agreement with interventions on the synthetic benchmark, where non-injected channels are pure noise. Could the authors run the corresponding channel- or lead-replacement intervention on the gate-repaired Sleep-EDFx or PTB-XL models, reporting agreement for both the latent-origin and the training-mean references? 4, The tabular example ties shape functions so that the aggregate is group-blind by construction. Can the authors show head or reference dependence of practical size on a standard NAM or NBM trained on a real task, with untied shape functions and deployable heads? 5.The real-data heads in Appendix N score near or below the majority-class rate on a 71%-wake prefix of the test set. Do the ranking differences across heads persist for heads that clearly outperform the majority baseline, evaluated on the full, subject-stratified test set?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.