- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how reliable SAE and crosscoder features are in understanding neural network behavior. They study this by removing a selected group of features, measuring the change in the model's output, then comparing it to a random intervention. This depends on how the intervention and its baseline is defined, so they consider several choices: how the features are encoded and removed, how large the intervention is, and whether the intervention preserves differences across inputs. Their experiments use two TabPFN versions, a public GPT-2 SAE, and dictionaries from Pythia checkpoints, demonstrating how experimental choices are critical to making conclusions about feature importance.
Critical strengths
- The paper's experiments are thorough and unbiased, showing that the choice of control can sometime reverse the result of interest, and that a larger ablation doesn't immediately mean greater feature importance.
Critical weaknesses
- The geometric bound is generously interpreted. The proposition does not prove that a population of matched controls cannot exist; the justification for this is unclear. This is presented as a central theoretical contribution but is overstated.
- Many parts of the paper, including figures, seem AI-generated with little human iteration. For example, the abstract contains various cryptic / unintelligible phrases ("Dose costs most", "the rest is dose", "admission alone barely improves a comparison’s sign") and even the names of packages "tabdiff.diagnostics.audit". Figure 1, which typically summarizes the paper's main result or contribution in a visually appealing way, is completely unreadable: it is not clear what the y-axes for each subfigure are, what the sub-figure in (1b) represents (it doesn't even have a labeled x or y axis), "91% artifact" clipping across the bar, and the caption certainly does not help. Even in the first couple of paragraphs, which should motivate the problem, the authors throw around various results that are meaningless without context: how do they measure disruptiveness and the 2.9x figure, the rho = -0.57 number, what the -0.461 deficit means, etc. These presentation issues, which continue throughout the rest of the paper, make it very difficult for a human to understand and assess the contribution of the paper. Therefore, the paper's current presentation does not meet the threshold for ICLR and should receive a clear rejection.
Questions for authors
Can the authors elaborate on the geometric bound, and why it establishes that a population of matched controls cannot exist? Could many distinct matched directions still exist near the dominant eigenvector? Please clarify what makes such a population unsuitable as a control and justify the threshold used to exclude it.