- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a mechanistically-motivated framework for pretraining data selection organized around three sequential bottlenecks: Circuit (the required computation, e.g., a binding-matching-transport read path, has not formed), Store (content is unavailable in parametric memory, split into M-Write and M-Use), and Use (content is available but the query-authorized route fails to control the output). A shared "diagnosis-to-data" principle - localize the missing operation, preserve its defining relation, vary shortcut-bearing context, re-audit the residual - derives stage-specific supervision.
Empirically: (i) a 1.485B / 100B-token matched-multiset run where a 1.2B-token Curriculum prefix (formation-time MDA + joint-to-matching order) crosses Ordinary's final validation PPL at 82B and retains the advantage through 100B; (ii) auxiliary 160M order studies showing domain-selective gains (Overall/Math/Code improve, Web does not); (iii) a Store 2x2 factorial (Write x Use) on 512 circuit-dragged facts showing Use contributes more than Write; (iv) paired-Use supervision improving strict-pair accuracy by 4.4pp over balanced-unpaired on familiar queries, and CCUR beating LongCE and a score-shuffled control for long-context window ranking; (v) a continuous 350M chain on 640 common facts where the complete Circuit-Store-Use sequence outperforms four stage-replacement controls. The paper reports negative results honestly: independent-query transfer fails, opposed-source arbitration remains open, and Web does not benefit from Circuit training.
Critical strengths
-
Experimental controls are substantially more rigorous than typical curriculum-learning papers. The matched-multiset design (positions change, data does not), the order study (joint-to-matching vs interleaved vs reverse), the exit sweep, three independent 3B initializations, and score-shuffled / document-disjoint replications for CCUR make the positive claims more credible. This matters because the headline effects are small (~0.43% relative PPL at 100B), and without matched comparisons they would be hard to distinguish from noise.
-
The Write/Use distinction in Store is a genuinely useful conceptual contribution. Separating "content not written" from "content written but suppressed by a natural carrier," and showing via a 2x2 factorial that Use training contributes more than Write for dragged facts (Table 24), has practical implications for knowledge editing and retrieval-augmented settings. The paper is also commendably transparent about where the framework fails (Web, independent-query transfer, CCUR's memory-authorized regression).
Critical weaknesses
-
The Use stage - the framework's terminal and most important stage - does not deliver on its promise. By the paper's own audits, paired supervision underperforms Ordinary on independently phrased queries across every configuration (Table 28), CCUR improves context-authorized decisions while worsening memory-authorized ones with no joint gain (Table 32), and "conditional arbitration" is explicitly left open. The positive Use result reduces to "matched-route engagement under familiar queries," a much narrower claim than the three-stage framing implies. Since the framework's value proposition is that diagnosing the current bottleneck tells you what to teach next, an unsolved terminal bottleneck leaves the chain incomplete and is the primary reason I lean toward rejection.
-
Effect sizes are small and statistical support is thin. The headline 100B result is a ~0.43% relative validation PPL difference from a single trajectory per arm; seed variance exists only for three 3B runs (1.00-1.58%), and its transfer to 100B is unclear. The Store factorial reports NLL gains without confidence intervals, and the 350M chain's absolute pair accuracy is 5.26% with no ceiling analysis. The novelty is also incremental: MDA supplies the Circuit signal and LongCE the long-context baseline, and the formation-vs-endpoint difference (Table 13: 3.79557 vs 3.82161 at 1.2B) is modest with no significance test.
-
Presentation density makes the paper hard to evaluate. The abstract and introduction are heavy with terminology defined only later (or never), the notation is extensive and mostly single-use, and nine pages of main text carry seven experimental threads so no result is fully digested. For a paper whose contribution is a unifying principle, the writing barrier substantially limits its usefulness to the community.
Questions for authors
-
Scale dependence. Have you observed the Circuit-Store-Use ordering at 7B+, even in a small probe? If induction-head formation timing shifts with scale, does the Curriculum prefix length need rescaling, and by what rule?
-
Statistical confidence at 100B. The 3B runs give 1.00-1.58% gain while 100B gives 0.43% from a single trajectory per arm. Is this shrinkage expected, and can you provide any seed-variance estimate at the 100B scale (e.g., from the dense late-training window in Table 17)?
-
CCUR's source prior. Table 32 shows CCUR improving context override while degrading memory reject. Is this inherent to the full-local response signal, or could balancing authority-mode exposure during selection mitigate it? This bears directly on whether the Use stage can be made to work.