- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies how training examples affect language models' ability to apply learned facts. The authors construct a synthetic task that asks models to use a learned fact about an entity to fill in a missing symbol in a sequence, and then classify the completed sequence. They compare training settings in which each entity's questions are paired either with the same answer or with different answers. Across the evaluated models, pairing an entity's questions with different answers improves performance on held-out application questions.
Critical strengths
- The paper describes its experimental setup and procedures in detail, supporting reproducibility.
Critical weaknesses
- The paper has substantial presentation issues. The writing is difficult to follow, and the experimental design, results, and conclusions are not connected through a clear narrative.
- The central idea appears insufficiently novel. Assigning the same answer to all application questions about an entity creates an obvious shortcut: the model can fit these examples by memorizing a fixed name-answer association. Showing that varying the answers improves generalization therefore seems unsurprising. Prior work has already discussed related issues [1,2,3]. I also find limited novelty in the experimental setting and methodology.
- The paper reads more like an experimental report than a study offering meaningful new insights. Despite the extensive experiments, it is unclear what we learn beyond the familiar issue of shortcut learning or how the findings advance our understanding of generalization.
[1] Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models.
[2] Physics of Language Models: Part 3.1, Knowledge Storage and Extraction.
[3] Physics of Language Models: Part 3.2, Knowledge Manipulation.
Questions for authors
- Are there any differences between the synthetic tasks in W1, W3, and W4? Are their examples drawn from the same distribution? What is the purpose for constructing separate worlds?
- Regarding lines 197-198: Why are the BF16 checks introduced at this point? How are they related to the surrounding discussion?
- Regarding 'Application improves without better recall': What relationship between application performance and recall is this comparison intended to establish? The improvement follows a change in training, while recall is another measured outcome rather than a demonstrated cause of the improvement. What insight does reporting these two outcomes together provide beyond describing the observed behavior?
- What is the purpose of the internal intervention experiments in Section 5? Why replace MLPs rather than other components, such as attention heads? How do these experiments relate to the paper's main claim?
- Regarding 'Better application can coexist with worse use of supplied inputs': This observation seems unrelated to the original motivation and main claim of the paper, but I find it interesting. Could the authors provide a deeper analysis of its underlying causes?
- Beyond the familiar observation that models can learn shortcuts, what additional insights or contributions does the paper provide?
- Why use an HMM rather than another rule-based method to construct the synthetic examples?