acceptodds
Under review as a conference paper at ICLR 2027

A Count, Not a Fraction: Hundreds of Examples to Build a Mechanism, Dozens to Trust It

Abstract

How much data does it take to install a mechanism in a language model? Measurements at pretraining scale say the answer is a number of examples rather than a share of the corpus, and they say it without saying why. This paper reproduces that law in a system small enough to write down every path the information can take. A 20M-parameter decoder is trained on a task whose attention mask keeps the query away from the source it must report. A shortcut to a visible copy of that source is open in most training examples, blocked in a chosen number of them, and open but carrying a wrong value in a chosen number more. We expected a relay. What forms instead is a one-hop cache, written into the only positions that both see the source and are read by the query, while the positions after them decode the answer and cannot move it. In a pilot version of the task, formation is governed by the count of examples that force it rather than by their share of the data, and the transition does not move when the dataset grows by an order of magnitude. A model written after the experiments, in which each route’s contribution is a product of a write and a read, reproduces this. An example the shortcut already answers trains both routes in proportion to the attention each receives, so it cannot shift the balance between them, and with gradient clipping the same accounting locates, after the fact, the one delivery schedule that breaks the law, which packs the forcing examples into too few updates. Once the cache exists, a much smaller count of corrupted examples flips which route wins a conflict; across dataset sizes that dose stays a count only when the learning rate decays at the end of training. That flip is a re-weighting applied to every input rather than a detector for the conflicting ones, so most of the answer changes hands while clean accuracy stays where it was.

Primary area:
interpretability and explainable AI
Keywords:
mechanics interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xqzBOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is interested in memorization in LLMs and its mechanism. The authors train 20M decoder models on toy tasks, where a fraction of the training examples are blocked. The authors then investigate the accuracy of the model on cache. The authors show that what matters more for learning is the number of blocked examples than the fraction.

Critical strengths

I found this paper too hard to understand.

Critical weaknesses

The main weakness of the paper is clarity. I found the manuscript unusually difficult to read, to the point that even after the abstract and introduction I did not have a clear understanding of the experimental setup, the main phenomenon being studied, or the precise claims being made.

The paper introduces substantial paper-specific terminology before explaining it clearly. Even the title relies on notions such as “cache” and “trusting the cache,” which are not standard in this context (and “cache” is particularly potentially confusing given the usual meaning of KV cache in language models).

The abstract and introduction contain many sentences that are difficult to interpret without already knowing the full setup. For example:

  • “Measurements at pretraining scale say the answer is a number of examples rather than a share of the corpus, and they say it without saying why.”
  • “This paper reproduces that law in a system small enough to write down every path the information can take.”
  • “We expected a relay.”
  • “That flip is a re-weighting applied to every input rather than a detector for the conflicting ones, so most of the answer changes hands while clean accuracy stays where it was.”
  • “The two doses of Figure 1...”

Terms such as “cache,” “relay,” “dose,” “formation,” and “trust” are important to understand the paper, but are introduced in a way that makes the reader reconstruct their meaning while also trying to understand the experimental setup.

In my view, the paper needs a substantially simpler presentation: first define the task and the two competing mechanisms concretely, then state the empirical observations, and only afterwards introduce the higher-level terminology.

I also have a significant concern about the scope of the conclusions. The experiments study a single ~20M-parameter model on a synthetic task with a hand-designed attention mask. The paper does not provide evidence that the reported count-based phenomenon persists at larger model scales or in more naturalistic settings. The authors themselves list the single model size and synthetic setup as limitations. As a result, I find it difficult to assess how much the proposed mechanism tells us about pretraining-scale language models more generally.

Overall, I do not think these issues can be adequately addressed in a rebuttal. The problem is not a small number of unclear statements, but the presentation and framing of the paper as a whole. A substantial rewrite would be required for the contribution to be evaluated reliably, and I therefore do not think the paper is ready for publication in its current form.

Questions for authors

N-A

3 comments

Dear Reviewer, as the discussion period ends soon, we would greatly appreciate it if you could let us know whether our response has addressed your concerns. We are happy to clarify anything further.

Dear authors,

Thank you for the revision.

I have re-read the abstract, and still find it impossible to understand for a human: it contains "blocked examples, corrupted examples, copy route, cache, formation, arbitration, pilot task, main task, verb, etc..." Most of those terms are individually understandable, but the abstract is still unclear about the work positioning, and I find it very confusing.

I maintain my original judgment that the paper needs a substantially simpler presentation, and that the revision here does not solve it.

Thank you for re-reading. You are right that the abstract still leans on our own vocabulary, and we accept that this is a real problem with the paper, not a matter of taste. Here is the paper in plain words, which is the order the camera-ready abstract and introduction will follow:

We train a small Transformer on a task where it can get the answer in two ways: copy it from a visible position, or read it from a few intermediate positions where it learned to store it. We then ask how many special training examples it takes for the model (1) to learn the stored route at all, and (2) to prefer it over copying when the two disagree. The answer is a fixed number of examples, not a fixed share of the data: a few hundred for (1), a few dozen for (2). We show the same holds for induction heads in an ordinary Transformer, and that it holds only when the rest of the training data has already been learned.

Every other term will be introduced only after this paragraph. We appreciate the push; it made the paper clearer even where it did not yet reach the standard you asked for.

Reviewer_LfraOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper conducts experiments with a 20M-parameter language model on synthetic tasks to investigate whether absolute example counts, rather than proportions, govern the formation and use of an internal cache. I find the experiments informative and the findings potentially valuable for understanding how training examples shape internal mechanisms. However, I am not yet convinced of their generality as a principle of language-model training, and not sure about their broader implications and impact.

Critical strengths
  • The experiments are well designed for investigating the mechanism under the specific setting.
  • The paper also provides an initial theoretical analysis that offers a possible explanation of the observed phenomenon.
  • The authors clearly acknowledge the current limitations of the study, including the restricted experimental scope and the discrepancies between the theoretical model and the actual training setup.
Critical weaknesses
  • I am uncertain about the generality of the findings and their broader implications for real-world language modeling training.
Questions for authors
  • How do the authors expect the findings to change with model size, and how to support this expectation beyond the 20M-parameter setting?
  • Figure 1 reports a formation threshold of 378 examples in the pilot task and a switching threshold of 56 in the main task; could the authors clarify the threshold to form the cache in the main task?
  • Figure 6 shows similar switching thresholds under the specific learning-rate decay schedule; have the authors checked whether this result also holds with other decay schedules?
1 comment

Dear Reviewer, as the discussion period ends soon, we would greatly appreciate it if you could let us know whether our response has addressed your concerns. We are happy to clarify anything further.

Reviewer_e973OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how a small number of special training examples can induce a Transformer to form and trust a new computational route. The authors construct a controlled retrieval task in which the model can answer either through a visible copy or through a cache stored in intermediate bridge positions. Blocked examples force the model to use the cache, while corrupted examples create conflicts between the copy and cache routes.

The experiments show that the model forms a distributed cache in the value states of the bridge positions. In the pilot setting, cache formation appears to depend more on the absolute number of blocked examples than on their fraction of the training set. Once the cache is formed, a smaller number of corrupted examples can cause the model to shift from relying on the copy route to relying on the cache under conflict. Through logit attribution, the authors argue that this shift reflects global reweighting of the two routes rather than a conditional switch triggered only by conflicting inputs. They further propose a two-route model to explain the different sample thresholds for formation and switching.

Critical strengths
  • Important question with a clear framing. The paper studies the data requirements for forming a new mechanism versus trusting that mechanism under conflict. This question has potential relevance to data mixtures, knowledge conflicts, and poisoning.

  • Clean experimental design with a strong mechanistic evidence chain. The controlled two-route setting separates formation from arbitration. The authors use patching, ablations, and key/value decompositions to causally localize the cache, rather than relying on probing alone.

  • Clear and insightful findings. The paper shows that forming the cache and shifting trust toward it are distinct processes: the former requires hundreds of forced-use examples, whereas the latter can be altered by fewer conflicting examples once the mechanism exists. Moreover, this change in internal reliance need not be reflected in clean accuracy.

Critical weaknesses
  • Lack of validation in natural or practical settings. My main concern is that the results are obtained in a very clean but synthetic setting: a single 20M model, a hand-designed attention mask, and a synthetic retrieval task. It remains unclear whether count-like data doses, route formation, or conflict arbitration also arise in natural-language pretraining, naturally emerging circuits, or more realistic knowledge-conflict settings. A corresponding experiment in a more natural setting would substantially strengthen the practical significance of the work—for example, studying induction heads or another more practical circuit, and testing whether targeted data can predict or accelerate its formation.

  • Missing a key related-work citation. The paper does not cite Chen et al., Mechanistic Data Attribution: Tracing the Training Origins of Interpretable LLM Units (ICML 2026). This work directly studies how specific training examples causally affect the emergence of interpretable heads.

Questions for authors

See Weaknesses.

1 comment

Dear Reviewer, as the discussion period ends soon, we would greatly appreciate it if you could let us know whether our response has addressed your concerns. We are happy to clarify anything further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.