acceptodds
Withdrawn from ICLR 2027

T^3: Training-Time Testing for Data-Constrained Pretraining

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

High-quality data for pretraining is limited and grows more slowly than compute, which makes it important to get the most out of a fixed corpus. Prior work improves generalization under fixed data with heavy weight decay, ensembling, distillation, and layer looping. We ask whether the training data can also serve as validation data for every update. We propose Training-time testing (): it splits each batch into two halves, takes a temporary descent step on the first half, evaluates the gradient of the second half at the adapted weights, discards the temporary step, and updates with the mean of the two gradients. Added to strongly tuned data-constrained recipes, lowers validation loss by 0.012 to 0.015 nats per token at 300M, 600M, and 1.4B parameters. On SlowRun, adding the step alone improves the top one-hour script by 0.010 to 0.012 within its time limit, and scripts built on it set the leaderboard records on both the 15-minute and one-hour tracks. On both benchmarks the gain persists after ensembling. Probes show that the gain requires a descent step tested on other tokens; the cosine between the gradients of different examples does not increase, and flatness alone is insufficient. In a nonlinear model of repeated training, we prove that at a common training-loss target, cross-example testing learns more shared signal and less sample-specific noise than ordinary training, yielding lower population risk.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language modesl, data contrained pretraining, generalization, meta gradient

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_s1kAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Training-Time Testing (T3), a framework for language-model pretraining under data constraints that uses part of the training data as validation data to modify each gradient step. Specifically, the authors formalize T3 as meta-learning with a first-order meta-gradient, and attempt to explain its working principle through a comparison with SAM and curvature measurements, measurements of gradient agreement between split batches, and an analysis of a simplified network. In experiments including a fixed DCLM corpus and SlowRun, T3 consistently reduces the validation loss of the pretrained models while reporting a small (+6–9%) overhead.

Critical strengths

(1) Motivation

The idea of improving pretraining performance in data-constrained settings by using training data as validation data is plausible and appears to have potential.

(2) Consistent Gain

In the experiments presented in the paper, T3 shows a consistent reduction in validation loss. Although the magnitude is small, it is reasonably acceptable given that the proposed method exploits relatively few components.

Critical weaknesses

(1) Failure to identify the mechanism

The authors introduce several approaches (gradient agreement, sharpness-aware minimization, a simplified network) to identify why T3 yields gains, but they fail to identify the overall mechanism of T3. Instead, the various experiments aimed at identifying the mechanism are scattered throughout the main text and the appendix and do not form a single coherent narrative. For the T3 method to be well supported, a robust theory or mechanism that supports the method should be presented in addition to the empirical results.

(2) Additional experiments needed

(a) Although T3 shows consistent loss reductions in the reported experiments, no comparison with similar methods is provided.

(b) The experimental results are limited to validation-loss reduction. As in Kim et al. (2025), which the paper cites, experiments are needed to determine whether the pretraining gains from T3 translate into gains on downstream tasks.

  • Kim, Konwoo, et al. "Pre-training under infinite compute." International Conference on Learning Representations. Vol. 2026. 2026.

(3) Incomplete exposition

(a) The Introduction presents the motivation clearly, but it does not sufficiently explain why this motivation is valid or how it connects to the T3 method.

(b) Section 2.1 introduces the T3 update, whereas Section 2.2 reads as one of the attempts to identify the working mechanism of T3 rather than as an explanation of how T3 operates. For the coherence of the main text, I suggest devoting Section 2.2 to content that explains more concretely how T3 operates (e.g., the algorithm block in Appendix A).

(c) As noted in W1, the analyses aimed at identifying the mechanism are scattered across several sections and the appendix, and the paper does not provide a consolidated analysis of what makes T3 effective.

(d) It is not sufficiently explained what Section 5 and Proposition 1 are intended to establish. The paragraph 'Why this improves generalization' explains how cross-example testing helps, but it does not clearly explain its connection to Section 5 and Theorem 1.

(e) Appendices E.4–E.6 contain only headings without content, and the author TODO page has been left on the last page of the manuscript.

Questions for authors

(1) To clarify how T3 works, please present sufficient conditions rather than a list of necessary conditions. This need not be theoretically complete; it is sufficient if it provides clear intuition or insight into how the method works. In addition, please refine the scattered experiments and analyses aimed at identifying the mechanism and organize them into a single consolidated analysis.

(2) Please provide comparative experiments with methods similar to T3, as well as downstream-task results for models pretrained with T3.

Reviewer_cHp7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose a new optimization method called T^3. T^3 works by splitting a training batch into support and query halves. The gradient calculated from the support half moves the parameters temporarily, at which point gradients from the query half are calculated. Then, at the original parameters, the support and query gradients are averaged and passed to the outer optimizer.

The method shows improvements on models of various sizes, evaluated on a fixed pre-training corpus, as well as on SlowRun. They do comparisons to SAM and show improvements in ensemble settings.

The authors perform various experiments to explain why their method works.

Critical strengths
  1. The method shows reasonable strong improvement, delivering around 1.2%-1.5% lower perplexity. It also transfers to ensemble settings, which already deliver low perplexity. It does all this while adding relatively little overhead adding between 6%-9%. The results also have confidence intervals and the paper explains tuning procedures and evaluation settings.

  2. The paper includes experiments on why the temporary step works and how it reduces validation loss. The paper also includes some theoretical analysis in Section 5 on why T^3 might improve generalization.

Critical weaknesses
  1. The method is exactly proportional to two-step Reptile displacement in the raw, full-parameter version. The authors do acknowledge this, and note it in Appendix A, but it makes the method less novel than is presented in the main text.

  2. The SAM comparison is quite weak. The only hyperparameters that were tuned seem to be the radius, and outer parameters were left fixed. In addition to that, SAM doesn't seem to be the main competitor and other closely related optimization algorithms are missing. Other methods such as 2 step Reptile would be interesting to compare against.

  3. The paper is hard to follow and the Appendix seems to contain many results but are not presented in an easy to understand fashion. In addition to that, some places are unpolished, and an AI generated TODO list was left in the Appendix. It seems that AI proposed experiments and interpreted results, going against what the authors disclose.

Questions for authors
  1. How does T^3 significantly differ from 2 step reptile. Which additions that T^3 makes are necessary for the improvements? How do they contribute to reducing validation loss?

  2. Can you compare against more methods than SAM?

Reviewer_GYShOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes T3 for data-constrained pretraining. Each batch is split into two halves: the model takes a temporary descent step on one half, evaluates the gradient on the other half at the adapted weights, restores the original weights, and then updates using both gradients. The method consistently improves validation loss in repeated-data pretraining and on the SlowRun fixed-time benchmark, with relatively small overhead.

Critical strengths
  1. The empirical study is very thorough. The paper includes paired seeds, multiple model scales, fixed-time comparisons, ensembles, independent data orders, and many useful ablations. The gains are fairly consistent at 300M–1.4B.

  2. The mechanism experiments are convincing. The improvement requires a descent step tested on other tokens; random/ascent directions and testing on the same tokens do not reproduce the gain. The paper also reports negative results rather than hiding them.

Critical weaknesses
  1. The algorithmic novelty is somewhat limited. T3 is closely related to FOMAML/MLDG, Reptile, and extragradient-style updates. The main novelty seems to be applying and validating this idea carefully in data-constrained LM pretraining, rather than introducing a fundamentally new optimization principle.

  2. The theory does not fully explain the empirical behavior. The theoretical model is quite simplified, and it predicts that self-testing should outperform ordinary training, while empirically self-testing performs poorly. The authors also acknowledge that the theorem does not directly cover Transformer training.

  3. The effective regime seems fairly specific. T3 works best for heavily regularized repeated-data recipes, but does not reliably help in long one-pass training and can lose its benefit under some recipe changes.

Questions for authors
  1. What is the main algorithmic difference between T3 and prior first-order meta-learning methods such as MLDG/FOMAML or Reptile?

  2. How do the authors interpret the gap between the theory and experiments, especially the opposite behavior of self-testing?

  3. Why does T3 depend so strongly on the underlying repeated-data/regularized recipe? For example, why does the gain largely disappear with weaker weight decay at 1.4B?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.