- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes RepInf, estimating which features of the training data affect LLM predictions. The main idea is applying an SAE to a middle layer, then calculating influence functions with respect to those features. By applying a JVP trick, the method is applicable to 1B LLMs. The experiment shows that SAE features have a higher stable rank than token or text representations. Ablating the top-scoring features affects the test prediction more than frequency- or magnitude-based feature selection. The method retains the influence score ranking of the training data, showing that ignoring the earlier parameters has little effect on the influence ranking.
Critical strengths
- Decomposing the training data attribution into SAE features is novel. Existing work mainly attributes data in terms of a single example or token. This paper proposes a new perspective and provides an efficient calculation method. The method can also be used when the loss of the data is not the sum of the token losses.
- The method does not reflect the effect of the parameters before the SAE layer. The paper provides the experimental result that shows the method retains the ranking of the original influence functions, showing this approximation does not hurt the influence score much.
Critical weaknesses
- The relation between the stable rank and attribution quality is unclear. The paper uses stable rank and related metrics to argue that SAE features are a more proper unit for data attribution. However, whether a high stable rank connects to better attribution quality is not shown. Stable rank may not be proper to measure the correlation of representations, since it is invariant to rotation.
- Although the paper states that feature-level attribution is their contribution, the paper does not validate whether their feature attribution uncovers important SAE features in training. RQ2 ablates the features on the test input; however, it does not show the effect of the features in training. RQ3 analyzes the effect in training; however, it just shows a data-wise analysis, not a feature-level analysis.
- The paper does not systematically analyze the top influential features their method discovers. To understand which part of the data affects LLM predictions, we should be able to interpret the top features the method discovers. The paper concentrates on providing case studies of which training data has a high score; however, showing which SAE features of that data make the high score is more needed for this method.
Questions for authors
- Could the authors explain the relation between stable rank and attribution quality?
- Could the authors provide an experiment that shows whether the discovered influential features are actually important in training?
- Could the authors provide a detailed case study of the top features?