acceptodds
Withdrawn from ICLR 2027

Tracing Influence: Unboxing LLM Outputs with Sparse Latent Spaces

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

17 views

Abstract

A critical step toward the reliable use of large language models (LLMs) in high-stakes domains such as healthcare is to attribute their predictions to training data, akin to a medical case study. This requires fine-grained precision: pinpointing not only which training examples influence a decision, but also which parts of them are responsible. While influence functions offer a principled framework for this, prior token-level approaches are restricted to losses that decompose additively across tokens and distribute credit over a strongly correlated token basis. We introduce \method, a framework that attributes influence to internal features through a latent mediation approach and is defined for any differentiable objective. Our method attaches a sparse autoencoder (SAE) to an intermediate layer of a finetuned LLM to obtain a sparse, less correlated attribution basis, derives an activation-weighted first-order attribution of the training gradient, and scales it with derivative swapping, reducing the per-feature Jacobian-vector products to a single additional reverse-mode pass (13–18 faster). Token-level influence is then obtained by projecting latent attributions back to the input space via token activation patterns. Experiments on multiple-choice QA with Llama-3.2-1B and Qwen2.5-1.5B show that influence-selected features are more necessary and sufficient for the prediction than activation-, frequency-, or randomly-selected features, including under on-distribution replacement, and are better than a gradientactivation baseline in several settings while never significantly worse. The total feature influence preserves the magnitude ranking of training examples under direct influence (Spearman on Qwen2.5-1.5B), and reweighting the selected training data shifts predictions more than reweighting random data, supporting \method as a practical tool for model auditing.

Primary area:
interpretability and explainable AI
Keywords:
LLM, Mechanistic Interpretability, Influence Functions, SAE

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9t98OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RepInf, estimating which features of the training data affect LLM predictions. The main idea is applying an SAE to a middle layer, then calculating influence functions with respect to those features. By applying a JVP trick, the method is applicable to 1B LLMs. The experiment shows that SAE features have a higher stable rank than token or text representations. Ablating the top-scoring features affects the test prediction more than frequency- or magnitude-based feature selection. The method retains the influence score ranking of the training data, showing that ignoring the earlier parameters has little effect on the influence ranking.

Critical strengths
  • Decomposing the training data attribution into SAE features is novel. Existing work mainly attributes data in terms of a single example or token. This paper proposes a new perspective and provides an efficient calculation method. The method can also be used when the loss of the data is not the sum of the token losses.
  • The method does not reflect the effect of the parameters before the SAE layer. The paper provides the experimental result that shows the method retains the ranking of the original influence functions, showing this approximation does not hurt the influence score much.
Critical weaknesses
  • The relation between the stable rank and attribution quality is unclear. The paper uses stable rank and related metrics to argue that SAE features are a more proper unit for data attribution. However, whether a high stable rank connects to better attribution quality is not shown. Stable rank may not be proper to measure the correlation of representations, since it is invariant to rotation.
  • Although the paper states that feature-level attribution is their contribution, the paper does not validate whether their feature attribution uncovers important SAE features in training. RQ2 ablates the features on the test input; however, it does not show the effect of the features in training. RQ3 analyzes the effect in training; however, it just shows a data-wise analysis, not a feature-level analysis.
  • The paper does not systematically analyze the top influential features their method discovers. To understand which part of the data affects LLM predictions, we should be able to interpret the top features the method discovers. The paper concentrates on providing case studies of which training data has a high score; however, showing which SAE features of that data make the high score is more needed for this method.
Questions for authors
  • Could the authors explain the relation between stable rank and attribution quality?
  • Could the authors provide an experiment that shows whether the discovered influential features are actually important in training?
  • Could the authors provide a detailed case study of the top features?
Reviewer_EZ9DOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

In this work the authors introduce the RepInf approach to influence attribution, which is an extension of the LLM architecture with an inserted SAE that contains sparse features. The work presents evidence that this injected latent space provides less correlated (though not orthogonal or independent) representations, which improve feature-based attribution. The authors present a formalism for decomposing the influence and validate it on feature pruning and empirical donor replacement.

Critical strengths
  1. Mathematical formalization of influence estimation through sparse internal representations
Critical weaknesses
  1. The introduction claims are misleading w.r.t. the provided experimental evidence. The expectation was to see more token-level data attribution ablations. It is mentioned briefly only at the end, while the majority of experiments target feature-level attribution. Token-level attribution is validated only by four case studies.

  2. The related work (Section 2) and, it seems, the models are dated from 2024.

  3. SAE insertion, even though cheap (as stated), requires a layer sweep and additional training, while inserting a reduction point into the architecture. The provided experiments do not demonstrate clearly that such a method is justified in comparison with the direct application of token-level influence functions (EK-FAC, DataInf, TracIn) on separate LoRA modules.

  4. The method is tried only on classification tasks; a broader study on autoregressive tasks is required.

  5. It is not clear whether attributions computed with the SAE transfer well onto the default model. Applying the model with the SAE in applications would be impractical due to the performance drop.

Questions for authors
  1. Analyses RQ1 (5.2) and RQ2 (5.3) represent a study of the injected features' selection and manipulation. I was interested in practical applications of the influence estimation method (like "Counterfactual reweighting"). It would be nice if you could elaborate more on different practical applications and compare them with the token-wise IF method (Formula 2) in such contexts.
Reviewer_EdmhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work studies token-level and representation-level influence for data attribution and data selection in finetuning stage. Typical influence function works focus on training token attribution. In contrast, this paper proposes integrating a sparse autoencoder into each layer to additionally attribute to intermediate layers in the model (RepInf).

Experiments use three question-answering datasets (OpenBookQA, CommonsenseQA, and MedQA) and covers to small transformers (Llama-3.2-1B-Instruct and Qwen-2.5-1.5B-Instruct). Experiments only include LoRA and no full parameter finetuning. The baselines used for attribution are not clear to me. From my understanding, there is no direct comparison of RepInf against existing token-based attribution methods. The paper doesn't include any experiments on data selection for post-training or pretraining, which is the standard downstream evaluation for influence functions.

Critical strengths
  • This paper moves beyond the traditional training data attribution using influence functions to capturing the relationship between latent features from intermediate layers to test performance.
  • Experiments show that RepInf outperforms the activation-based baselines.
Critical weaknesses
  • The experiments section could be significantly improved. For instance, how does the proposed latent feature influence affect downstream tasks. For training data attribution, how does it compare to methods such as EK-FAC and TracIn? Can the proposed method be further used for data selection in finetuning stages?
  • The paper presentation makes it really hard to follow key results, especially section 5 (Experiments).
  • Its not clear how this methodology scales to models beyond 1B parameters? I believe this is very important to do interpretability analysis on any real-world systems.
Questions for authors
  • Figure 3 needs reformatting.
  • Abstract is really hard to follow and could benefit from a rewrite.
  • Some results are only mentioned inline (section 5.5) and many experimental results are pushed to Appendix without clear indication (e.g., line 434). I'd recommend editing section 5 to keep core results in the main paper text.
  • The argument starting in line 210 is not clear to me, could you clarify?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.