- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper is interested in memorization in LLMs and its mechanism. The authors train 20M decoder models on toy tasks, where a fraction of the training examples are blocked. The authors then investigate the accuracy of the model on cache. The authors show that what matters more for learning is the number of blocked examples than the fraction.
Critical strengths
I found this paper too hard to understand.
Critical weaknesses
The main weakness of the paper is clarity. I found the manuscript unusually difficult to read, to the point that even after the abstract and introduction I did not have a clear understanding of the experimental setup, the main phenomenon being studied, or the precise claims being made.
The paper introduces substantial paper-specific terminology before explaining it clearly. Even the title relies on notions such as “cache” and “trusting the cache,” which are not standard in this context (and “cache” is particularly potentially confusing given the usual meaning of KV cache in language models).
The abstract and introduction contain many sentences that are difficult to interpret without already knowing the full setup. For example:
- “Measurements at pretraining scale say the answer is a number of examples rather than a share of the corpus, and they say it without saying why.”
- “This paper reproduces that law in a system small enough to write down every path the information can take.”
- “We expected a relay.”
- “That flip is a re-weighting applied to every input rather than a detector for the conflicting ones, so most of the answer changes hands while clean accuracy stays where it was.”
- “The two doses of Figure 1...”
Terms such as “cache,” “relay,” “dose,” “formation,” and “trust” are important to understand the paper, but are introduced in a way that makes the reader reconstruct their meaning while also trying to understand the experimental setup.
In my view, the paper needs a substantially simpler presentation: first define the task and the two competing mechanisms concretely, then state the empirical observations, and only afterwards introduce the higher-level terminology.
I also have a significant concern about the scope of the conclusions. The experiments study a single ~20M-parameter model on a synthetic task with a hand-designed attention mask. The paper does not provide evidence that the reported count-based phenomenon persists at larger model scales or in more naturalistic settings. The authors themselves list the single model size and synthetic setup as limitations. As a result, I find it difficult to assess how much the proposed mechanism tells us about pretraining-scale language models more generally.
Overall, I do not think these issues can be adequately addressed in a rebuttal. The problem is not a small number of unclear statements, but the presentation and framing of the paper as a whole. A substantial rewrite would be required for the contribution to be evaluated reliably, and I therefore do not think the paper is ready for publication in its current form.
Questions for authors
N-A
3 comments
Dear authors,
Thank you for the revision.
I have re-read the abstract, and still find it impossible to understand for a human: it contains "blocked examples, corrupted examples, copy route, cache, formation, arbitration, pilot task, main task, verb, etc..." Most of those terms are individually understandable, but the abstract is still unclear about the work positioning, and I find it very confusing.
I maintain my original judgment that the paper needs a substantially simpler presentation, and that the revision here does not solve it.
Thank you for re-reading. You are right that the abstract still leans on our own vocabulary, and we accept that this is a real problem with the paper, not a matter of taste. Here is the paper in plain words, which is the order the camera-ready abstract and introduction will follow:
We train a small Transformer on a task where it can get the answer in two ways: copy it from a visible position, or read it from a few intermediate positions where it learned to store it. We then ask how many special training examples it takes for the model (1) to learn the stored route at all, and (2) to prefer it over copying when the two disagree. The answer is a fixed number of examples, not a fixed share of the data: a few hundred for (1), a few dozen for (2). We show the same holds for induction heads in an ordinary Transformer, and that it holds only when the rest of the training data has already been learned.
Every other term will be introduced only after this paragraph. We appreciate the push; it made the paper clearer even where it did not yet reach the standard you asked for.
Follow-up
Dear Reviewer, as the discussion period ends soon, we would greatly appreciate it if you could let us know whether our response has addressed your concerns. We are happy to clarify anything further.