- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper explores the reliability of sparse autoencoder attribution graphs under feature interventions. It shows that existing fixed-support graphs often fail to account for changes in active features after interventions. To address this, the paper proposes Margin Graph, which combines first-order approximations with TopK re-ranking to predict downstream feature changes.
Critical strengths
None
Critical weaknesses
-
The presentation of this work is poor. The paper contains numerous awkward and unclear sentences. The overall organization also lacks a clear focus, making it difficult to identify the main contributions. In particular, the theorem statements and figure captions are unnecessarily lengthy and difficult to follow. The paper needs substantial improvement in writing, structure, and clarity. The AI Use Statement of this work also indicates extensive use of generative AI tools throughout the research and writing process.
-
The technical novelty is limited. The proposed Margin Graph essentially applies a first-order approximation followed by TopK re-ranking, which is closely related to a single-step gradient approximation (e.g., IG with k=1). Moreover, it does not outperform the compared re-ranked baselines, including IG, in prediction accuracy. The paper therefore does not clearly establish a substantial methodological advance over existing techniques.
Questions for authors
No specific questions. The main concerns relate to the limited technical novelty and very poor presentation, which are unlikely to be resolved through clarification alone.
4 comments
Follow-up Comment to Reviewer pb63
We further revised the paper again to make it easier to read. The changes are listed below:
- The abstract and introduction now explain features, coefficients and selection before they introduce the margin graph.
- Section 2 gives the definitions, and Table 2 collects the symbols in one place. Where the words preactivation, active, inactive and code named measured quantities, we replaced them with encoder score, selected, unselected and coefficient vector, respectively.
- The introduction now lists six findings.
- Figure 1 now illustrates selection before and after an intervention and marks the cutoff and one feature's margin. It no longer contains the measurements for the contributions described in our first response.
- Plain explanations now come before the main theorems, and the captions define their quantities and populations.
- Our earlier statement that every theorem and caption met the short length limits no longer holds for this version. Some statements and captions are now longer, so that their objects and conditions can be read alongside them. The mathematical conditions and conclusions are unchanged.
- Each results section explains the purpose of its comparisons.
- Section 4 still distinguishes selecting features again from improving the derivative approximation, and its comparison keeps the stronger reranked baselines. Figure 3 now sits in Appendix B.
- Section 7 states the relation to attribution patching and integrated gradients, and Section 5 keeps the adverse result for the measured model output.
No experiment or numerical result has changed since last version, and the experiments still use the same models, SAE pairs and task.
Thank you again for your time and consideration.
Thanks for the authors' response. The writing is better than before, but the presentation and organization still need considerable improvement. In addition, the comparison with baselines also does not show a clear advantage of the proposed method. Therefore, I will maintain my score.
Thank you for reading the revision. We always agree that reranked integrated gradients achieves lower coefficient error than the margin graph, and we also agree with your observation about one-step approximations: evaluating the derivative at the clean endpoint and then applying TopK gives our predictor. Section 7 acknowledges the prior work behind this order of operations.
The comparison we would most like to draw your attention to is within the softmax baseline itself. When one feature is zeroed on the public Pythia-70M pair at layers 3–4, the median relative coefficient errors are:
| Score approximation | Keep the original selection | Select again from all predicted scores |
|---|---|---|
| First order | 0.739 | 0.121 |
| Softmax correction | 0.731 | 0.072 |
These four cells use the same interventions (Table 15, p. 33). Improving the score approximation while keeping the original selection changes the error from 0.739 to 0.731. With that same softmax correction, selecting again changes it from 0.731 to 0.072. The stronger baseline is therefore particularly informative: it shows why improving the score approximation alone leaves much of the coefficient error in place. Across the five public TopK pairs, zeroing one feature changes the downstream selection in 74–89% of cases (§3).
We appreciate that you may judge this empirical finding insufficiently significant for ICLR. Could you clarify whether that is your remaining concern, or whether you see a particular problem with the evidence for the finding? If a passage in the revised paper makes that evidence difficult to assess, a specific example would help us address it.
Thank you again for your time.
Comment to Reviewer pb63
Thank you for your time on our paper and for your candid review. We have revised the paper for clarity and flow.
1. Presentation, sentences and focus. Now, the main text follows one claim, that an attribution graph between TopK sparse autoencoders (SAEs) is decided at the margin. The introduction closes on four numbered contributions, and Figure 1 draws the method above one measurement for each contribution.
2. Theorem statements and captions. Each theorem in the main text now states its hypotheses and its conclusion within five lines, and a paragraph after it says what it means. Each caption now opens on the finding its exhibit shows and stays within sixty words, or within a hundred where its symbols or counts need definitions. The theorem statements and the captions describe the objects and the comparisons they present.
3. The statement on the use of language models. The statement is our honest disclosure under the venue's policy, and we respectfully suggest that it might not be counted as a critical weakness.
4. Technical novelty and integrated gradients at one step. The margin graph does apply a first-order approximation and then rerank. Section 4 now compares it with the reranked corrections and separates clean derivatives that can be reused from work that is repeated for each intervention. We measured integrated gradients at five and at ten JVPs and not at a single step, so the page reports no number for one step. The margin graph takes its JVPs at the clean input, so one set of JVPs at a position serves every intervention at that position, at every scaling factor and on every subset. A path average must be computed again for each intervention.
5. Accuracy against the reranked corrections. On our SAEs, reranked integrated gradients err by 8.8 × 10⁻⁶ at five JVPs per intervention and the reranked softmax correction errs by 0.013, against 0.017 for the margin graph at one JVP. The reranked softmax correction errs less than the margin graph on all five public pairs. Section 4 reports this as a finding and prints each method's JVP count beside its error. On the clean active set, every correction errs by 0.777 to 0.783 against 0.786 for the standard graph, so reranking removes most of the error and the first-order correction removes little of it.
6. The methodological advance. The advance the page claims is the measurement around a known form. On five public TopK pairs, zeroing one feature moves a downstream feature across the cutoff in 0.74 to 0.89 of cases. Theorem 1 proves the margin graph exact on an affine segment. Section 4 finds that reranking removes most of the error, and the margin graph reranks at the JVPs the standard graph already takes.
7. Concerns beyond clarification. The general response states the page's claim, its behavioral result and its measured costs, and it names the sections that establish each of them.