- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors test the model on two tasks, Operator Induction and Fake Word Inference, on 4 open-source models, and finds significant performance drop after corruption. This paper discovers that under corruption, models encode both correct and incorrect rules in intermediate layers but develop prediction confidence only in late layers. And masking specific head would significant mitigate this issue!
Critical strengths
-
- This paper finds an interesting and important problem, where the model's performance would significantly drops after some simple corruptions, such as conflicting demonstrations,
-
- This work designs two interesting test tasks, Operator Induction and Fake Word Inference, to test the robustness of LLMs.
-
- The authors find out two kinds of attention heads, which are Vulnerability Heads and Susceptible Heads, specifically related to the degrading performance. And experiments shows that masking them would lead to significant performance improvements.
Critical weaknesses
-
- The limited scope of tested LLMs. The authors only tested the model up to 8B and limited to two small model families (Qwen and Llama). Expanding to more open source and even proprietary models would significantly strengthen the claim.
-
- The presentation of the paper is not straightforward and hard to comprehend, especially for diagrams and figures!
-
- The performance was reported on single tasks, and not on some public benchmarks or compared with baselines. The performance on the public benchmarks like GSM8K or MATH would significantly strengthen the generalizability of the claim.
Questions for authors
-
- Halawi et al. 2023 also mentioned the similar findings that some attention head is highly relevant to the worse performance with low-quality demonstrations. Though the paper cites the work, and claim that Halawi et al. 2023 mainly focuses on text QA and classification. I'm curious what's the difference between the submitted work with the related work other than task difference.
-
- For table 1, I'm curious the performance without prompt corruption!
-
- There are major inconsistencies in the paper, such as mismatch between Sec. 5.2 and Fig.12. The example in Sec. 5.2 for top susceptible heads are L30H29 和 L31H5, but in Fig. 12 are L34H18 and L33H31. -4. There are a lot of grammar errors in the paper.
3 comments
Thanks for the detailed rebuttal! It addresses most of my concerns but I still remain concerns about the generalization of the method. Therefore I would like to keep my score. Best wishes for the paper!
Authors Comment
Dear Reviewer 8Mst,
Thank you again for your constructive suggestions and kind words! We are glad that our response addresses most of your concerns. Your suggestions have made the paper clearer and its mechanistic analysis more complete.
We agree that broader generalization is important, and Section 6 names larger models and richer rule-inference settings as future work. As an initial mechanistic analysis, the paper finds the same failure on two tasks and traces it to the same two head families in four models from two families, with consistent evidence from probes, the logit lens and targeted ablation. We hope this work encourages the community to test the mechanism in larger models and more tasks.
Best regards,
The Authors
Authors Comment
Dear Reviewer 8Mst,
Thank you for your review. We would appreciate hearing whether our response addresses your concerns, and are happy to clarify anything further.
Best regards,
The Authors