- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes Training-Time Testing (T3), a framework for language-model pretraining under data constraints that uses part of the training data as validation data to modify each gradient step. Specifically, the authors formalize T3 as meta-learning with a first-order meta-gradient, and attempt to explain its working principle through a comparison with SAM and curvature measurements, measurements of gradient agreement between split batches, and an analysis of a simplified network. In experiments including a fixed DCLM corpus and SlowRun, T3 consistently reduces the validation loss of the pretrained models while reporting a small (+6–9%) overhead.
Critical strengths
(1) Motivation
The idea of improving pretraining performance in data-constrained settings by using training data as validation data is plausible and appears to have potential.
(2) Consistent Gain
In the experiments presented in the paper, T3 shows a consistent reduction in validation loss. Although the magnitude is small, it is reasonably acceptable given that the proposed method exploits relatively few components.
Critical weaknesses
(1) Failure to identify the mechanism
The authors introduce several approaches (gradient agreement, sharpness-aware minimization, a simplified network) to identify why T3 yields gains, but they fail to identify the overall mechanism of T3. Instead, the various experiments aimed at identifying the mechanism are scattered throughout the main text and the appendix and do not form a single coherent narrative. For the T3 method to be well supported, a robust theory or mechanism that supports the method should be presented in addition to the empirical results.
(2) Additional experiments needed
(a) Although T3 shows consistent loss reductions in the reported experiments, no comparison with similar methods is provided.
(b) The experimental results are limited to validation-loss reduction. As in Kim et al. (2025), which the paper cites, experiments are needed to determine whether the pretraining gains from T3 translate into gains on downstream tasks.
- Kim, Konwoo, et al. "Pre-training under infinite compute." International Conference on Learning Representations. Vol. 2026. 2026.
(3) Incomplete exposition
(a) The Introduction presents the motivation clearly, but it does not sufficiently explain why this motivation is valid or how it connects to the T3 method.
(b) Section 2.1 introduces the T3 update, whereas Section 2.2 reads as one of the attempts to identify the working mechanism of T3 rather than as an explanation of how T3 operates. For the coherence of the main text, I suggest devoting Section 2.2 to content that explains more concretely how T3 operates (e.g., the algorithm block in Appendix A).
(c) As noted in W1, the analyses aimed at identifying the mechanism are scattered across several sections and the appendix, and the paper does not provide a consolidated analysis of what makes T3 effective.
(d) It is not sufficiently explained what Section 5 and Proposition 1 are intended to establish. The paragraph 'Why this improves generalization' explains how cross-example testing helps, but it does not clearly explain its connection to Section 5 and Theorem 1.
(e) Appendices E.4–E.6 contain only headings without content, and the author TODO page has been left on the last page of the manuscript.
Questions for authors
(1) To clarify how T3 works, please present sufficient conditions rather than a list of necessary conditions. This need not be theoretically complete; it is sufficient if it provides clear intuition or insight into how the method works. In addition, please refine the scattered experiments and analyses aimed at identifying the mechanism and organize them into a single consolidated analysis.
(2) Please provide comparative experiments with methods similar to T3, as well as downstream-task results for models pretrained with T3.