- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies whether information that can be linearly decoded is actually used by the model. It shows that parity can be decoded from pretrained embeddings even though the model cannot answer parity questions. After fine-tuning on an arithmetic verification task, the model mainly learns a parity shortcut instead of arithmetic. The later experiments show that this behavior is brittle and depends on specific tokens and formats rather than an abstract parity rule.
Critical strengths
- Control setups are used so shortcut can be identified clearly.
- The parity-neutral and out-of-distribution tests make it easy to see what the fine-tuned model is doing.
- The paper includes several diagnostic and causal controls instead of relying only on probe accuracy.
Critical weaknesses
- The paper is a technical report written by AI that is mostly unreadable. There are massive paragraphs with dense numerical performance numbers, repetitive language, and little to no structure in the paper.
- The easy and difficult settings vary in arithmetic complexity, but both preserve the same perfect parity shortcut. As a result, the main finding still depends strongly on the way the training data is constructed
- The paper overinterprets probe accuracy as evidence of a parity representation. Its own analysis shows that parity is mostly recoverable from final-digit identity, which is a lexical property. Prior work has already shown that lexical information can dominate representations [1], probe accuracy does not necessarily imply task relevance [2], and decodability does not establish causal use [3].
- There are no insights in the paper that are not already well known from probing to shortcut learning.
[1] On the Persistent Effects of Lexicality in Large Language Models
[2] Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?
[3] Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals
Questions for authors
N/A