- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper utilizes the perturbation method, rotating toward the feature direction of the residual vector in an early layer, and measures the effect via distance metrics such as L2 and cosine similarity in a later layer. The results on feature-specific direction and random direction illustrate that contrastive, MELBO, and top-activating SAE directions are privileged, consistent with superposition and error correction across six model families.
Critical strengths
- A direct and simple way to perturb the residual vector to dig out the feature effect
- This paper empirically illustrates the role of superposition in connecting multiple features to increase the robustness of the activation plateaus.
- Propose a sensitivity measurement of p for the feature perturbation effect.
Critical weaknesses
- The main concern is that the L2 distance can indicate the connection between the feature and the residual vector. However, the feature's magnitude in the final response is insufficient. In other words, the chosen feature may not have enough effect on the results.
- The organization of the paper is chaos, the meaning and the motivation for the methodology are unclear and make it hard to understand and to follow.
- The AI-written content introduces a bunch of redundant descriptions. For example, Figure 1 includes too many details.
Questions for authors
NA
3 comments
Thanks for your response. The W2 covers the methodology section. If you provide a clear structure, such as subsections, readers will find it easier to follow. The other questions are resolved, and I will keep my score.
Thank you for following up. We are glad to hear that the other points you have raised are addressed. Following your suggestion regarding the methodology section, we've broken it up into five subsections: 3.1 Perturbation and downstream response, 3.2 Plateau-breaking angle, 3.3 Direction types (further split into candidate feature directions and non-feature baselines), 3.4 Perturbations along two directions, and 3.5 Response model and superellipse fit. We also removed some redundant sentences in the same section. We've uploaded the revised PDF with changes marked in blue. We thank you for this suggestion and we hope the revision makes the paper easier to follow. If this addresses your concern, we would be grateful if you would consider updating your score.
Thank you for the review.
W1: L2 distance may not reflect the feature's effect on the final response. We are not sure we understand the point raised by the reviewer, but our best guess is: L2 distance at the second-to-last layer shows that a perturbation changes internal activations, but not that it changes the model's output. If this is the case, the submitted version of the paper addresses it in two ways. First, the response-metric ablation measures KL divergence of the output logits directly, and contrastive pairs still give p > 2 (Fig. 3, "Response metric": median 2.20 [2.10, 2.36], 81% of pairs above 2). Second, Appendix E (Fig. 6) shows that steering along these directions and their combinations visibly changes the model's completions. If the reviewer meant something else, we'd be glad to address it.
W2: Organization and motivation. We are sorry to hear the paper was hard to follow. If the reviewer can point to specific sections that were unclear, we'll be happy to revise them.
W3: Redundant descriptions; Figure 1 has too much detail. We simplified the Figure 1 caption in the revision.