- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper argues that different relationships between inputs can produce the same internal model representations and, consequently, the same sparse codes learned by a sparse autoencoder (SAE), since the SAE observes only model activations. To address this limitation, the authors introduce the optimal input scope curve, which characterizes the trade-off between a feature-scope budget (the number of input variables a feature is allowed to depend on) and the activation variance that remains unexplained under that constraint. They show theoretically and empirically that, as the arity of input interactions increases, a larger scope budget is required to achieve the same approximation tolerance. Overall, the paper argues that evaluating SAE features requires examining not only the model’s internal activations, but also the relationship between those activations and the model’s inputs. For example, activation-based metrics alone cannot determine how localized or input-dependent a feature truly is.
Critical strengths
-
The paper raises an interesting point that, to my knowledge, has not been explored in the field: different relationships between inputs can produce the same internal model representations. This suggests that SAE evaluation should also consider the mapping from inputs to internal representations, rather than relying solely on activation-based metrics.
-
I value the authors’ theoretical approach, especially because much of the existing work in this area is primarily empirical. The theoretical analysis and carefully constructed toy setting clearly isolate the distinction between representation-level equivalence and input-level dependence, making the paper’s central argument precise.
Critical weaknesses
-
The paper shows that different input-to-representation mappings can produce the same distribution of internal activations, but it does not demonstrate that these mappings lead to different model behavior or materially different interpretations in realistic LLMs. In fact, I would be surprised if they did: for a fixed downstream network, the same activation distribution at a given layer should induce the same output distribution. If the model treats the inputs equivalently at the representation level, it is unclear why their input mappings should be distinguished for interpretability purposes. The authors should therefore establish that the proposed input-scope differences correspond to different input-conditional behavior or other practically meaningful distinctions in realistic LLMs.
-
The paper is difficult to follow: I had to read the abstract and introduction several times before understanding the main claim. It moves quickly into technical details without first giving a clear motivation or an intuitive example of why the proposed distinction matters, which is especially problematic given the practical-relevance concern raised above. In addition, the related-work discussion is somewhat limited for a paper proposing a new SAE evaluation metric. It would be helpful to explain more explicitly how input scope differs from or complements existing concepts and metrics, such as feature absorption and feature disentanglement in SAEBench, or how it could provide insight into phenomena such as feature instability for example.
Questions for authors
-
Can the authors provide evidence that the proposed input-scope differences occur in realistic LLMs, rather than only in the constructed and controlled settings?
-
If two input mappings produce the same activation distribution, what practical interpretability or behavioral consequence justifies treating them as different?
-
How does the input scope curve differ from or complement existing SAE evaluation methods beyond reconstruction and sparsity?
Dear Reviewer QePF,
We would like to respectfully offer a reminder and invite you to join the further discussion. We believe that most concerns have been reasonably addressed or substantially resolved. If you have a different view, we can give further clarification. Alternatively, we would also welcome you to raise the score or adjust the confidence.
We would also like to share important new experimental results that may change your view of the paper. To address your central concern about practical interpretability and behavioral relevance, we conducted additional experiments connecting SAE input scopes to actual model output behavior.
We have also substantially revised the presentation and clarified how input scope complements existing SAE evaluation methods. Our responses have been reorganized to address your original questions directly.
Full experimental details are provided in our responses and revision. We look forward to your reply and will be happy to answer any further questions.