- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper formalizes in-context factual recall as identifying a hidden relation from subject–answer examples and then retrieving the answer to a new subject from an MLP associative memory. It analyzes fine-tuning dynamics in a one-layer transformer and reports synthetic experiments with constructed and pretrained memories.
Critical strengths
The paper isolates a useful question: how contextual examples select facts already stored in model parameters. The analysis gives a concrete attention mechanism and a sample-efficiency result under its stated assumptions.
Critical weaknesses
The main theoretical setting is far from the behavior the paper aims to explain. It assumes bijective relations, orthogonal token embeddings, an effectively complete frozen associative memory, and fine-tuning restricted to a positional attention block. The appendix reports a persistent generalization gap for a pretrained two-layer transformer. This substantially limits what the result establishes about factual recall in practical models.
The mechanism claim is stronger than the general theorem. The exact pairwise attention result comes from the three-subject analysis; Section 3.2 states that the near-optimal solution for general subject sets does not exhibit the exact pairwise pattern. The broader claim therefore rests largely on synthetic observations.
Questions for authors
Can the authors quantify when the pairwise pattern emerges in the general setting, and explain how the proposed mechanism accounts for the two-layer model’s generalization gap in Appendix A? Evidence on that point would materially affect my assessment.
2 comments
I thank the authors for their detailed response. After considering their clarifications, I have decided to maintain my original score.
We thank the reviewer for the review.
Our theoretical claims concern a one-layer transformer, as stated in the abstract and theorem assumptions. Please see our global response for the scope of the paper. The two-layer experiments in Appendix A explore a possible extension; we do not claim that our theory explains their behavior.
As explained in Section 1.1, we prove exact pairwise convergence in the three-subject setting and sample-efficient recall in the general setting. We also observe pairwise attention empirically beyond the three-subject setting.