- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work proposes an interventionist approach to identifying minimal sets of computation components, at the token-layer-level, which are key to an LLM’s answer output. These sets are shown to be effective at a smaller size than importance-based methods, and the methodology is shown to be general to many types of question-answering tasks.
Critical strengths
- The paper is exceptionally well-written and clear
- The experimental procedure is rigorous and reproducible
- The argumentation and analysis is convincing
- The claims made by the paper are careful and exact
Critical weaknesses
- Impact: results presented show that the importance baseline can pass the dual test for some (larger) position allowance. If I care only that I find some passing set and not that it is minimal (i.e. I care only about recall), then the presented method is not preferable to the baseline
- While in the strengths I said that the experimental procedure is rigorous (which it largely is), the importance baseline used was a simplistic one. The work’s own related-works section outlined a variety of methodologies, but only DLA was used as comparison.
Questions for authors
- “For real text we enlarge the null pool to include uncontaminated tokens before the answer span, since answer-span positions alone sit so close to the cut that random draws would nearly reproduce it.” I’m not sure I understood this. Real text as opposed to what? What is an answer span? If the cut is 2-3 token-positions, and the input text is at least a couple sentences, how could random 3-token subsets of the input text reproduce the chosen cuts?
- Is there any evidence to indicate that the representation at a given token position in a middle layer is actually associated with that position in the input? That is to say, if at layer l, if position s is found to be in the cut then does this mean that the information at position s of the input is a key piece of evidence? Since the representation at a given token position is a function of the previous representations at all token positions, I don’t think this is the case. While this may still allow for claims e.g. “the model concentrates information into a representation bottleneck”, it may not allow for explainability claims such as “we can identify which pieces of the input are key to the model’s answer”. Did you find that identified cuts overlapped with key pieces of information in your synthetic data experiments?
Thank you for your reply.
I thank the authors for a detailed and partly self-correcting response. As the authors have accepted and corrected W2, I will increase my score. However, I still have some concerns about W3, as the head-level comparison rests on DLA alone, and the causal baselines are position-level only.