- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper focuses on the geometry of the Linear Representation Hypothesis, specifically its implications of how activations should decompose into the tangent and normal spaces of the representation manifold. They show three key findings: the tangent component of activations contains most of the interpretable information measured by the J-lens, a theoretical analysis that shows J-lens can be modified without changing on-manifold model behavior and lastly empirical evidence that J-lens’s performance relies more on its normal component.
Critical strengths
-
The paper is well motivated, providing a new analysis of the linear representation hypothesis. The use of the projection to tangent and normal spaces is interesting and may provide motivation beyond this work.
-
The discrepancy between the interpretable information being tangent to the activation manifold and J-lens’s greater use of the normal space is interesting, and provides insight into a new method that is less understood and may have implications for future research.
Critical weaknesses
-
Given that the results show that the push-forward and PCA approximations of the tangent space capture different subspaces and in 4.3 had differences in results (Tables 1 and 2), it would be more convincing to see for section 5.1 the results with the PCA approximation as well. Seeing consistent trends across both methods would strengthen the claim of 5.1 and show it’s not an artifact of the tangent space approximation.
-
In Section 4.2, it is shown that the pushforward and PCA approximations capture substantially different subspaces. Therefore, it is hard to understand from the paper how well the approximations represent the tangent space, and by extension the reliability of the empirical findings.
Questions for authors
Both approximation methods contain parameters such as neighborhood size and energy (for PCA). How sensitive are the empirical results to the choice of hyperparameters? Also, the dataset used for generating activations might have an effect on the quality of the approximation, have the authors tried increasing/decreasing the number of sequences to validate their choice?
Hello! We just wanted to follow-up briefly, to see if you've had a chance to review the updated material. The deadline is in ~2 days, and we'd love to know if there are any more improvements we can make, or if we've satisfactorily resolved any reservations you had!