acceptodds
Under review as a conference paper at ICLR 2027

TIMELOOP: Turning Training Trajectories into Realized Future Supervision

Abstract

Training trajectories are usually consumed as optimization traces, even though their delayed intervention consequences contain information for later decisions. We study whether those consequences can become reusable empirical supervision rather than predictions of an unobserved future. TIMELOOP stores matched-state branch outcomes as a Future Memory whose records bind a training state, action, future horizon, and realized utility. At a later query, it retrieves comparable realized futures to rank candidate data mixtures before new outcomes are revealed. Across 20 natural decision states, Full TIMELOOP recovered the realized best action in 20/20 cases, whereas removing future supervision or using an Always-Balanced policy recovered 10/20. A natural signal-attribution study further found 22/22 top-1 decisions and mean Spearman correlation 0.9818 for Full TIMELOOP, compared with 54.55% top-1 for state-only, future-only, and randomized controls. In a controlled shift, old memory without aligned new evidence achieved 3/20, while progressively adding shift-aligned experience increased mean top-1 accuracy from 0.15 to 1.00 across five acquisition orders. These results support an experience-conditioned decision mechanism: realized training futures can become reusable supervision, but TIMELOOP is not a zero-shot oracle for unseen utility changes. The evidence is bounded to one 336M model, a controlled four-domain ecosystem, and five hand-defined actions; reliability also depends on memory support, horizon matching, action coverage, anchor isolation, and the cost of acquiring future branches.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model pretraining, pretraining data selection, adaptive data mixtures, training trajectories, realized future supervision, retrieval-based supervision

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_1RCDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TIMELOOP, a framework that reuses the observed consequences of previous training interventions to guide subsequent data-mixture decisions. It constructs a “Future Memory” containing training-state descriptors, candidate actions, evaluation horizons, and realized utilities obtained from controlled training branches. At a new query, an action-conditioned nearest-neighbor estimator predicts each action’s utility and selects the highest-scoring action. Experiments consider four training domains and five predefined mixture actions. The authors report strong action-selection accuracy, investigates the contribution of state and outcome information, and studies memory size, horizon matching, and adaptation after acquiring evidence aligned with a changed utility function.

Critical strengths
  1. The problem is relevant, and the intervention-based data collection is well motivated. Branching from a common model and optimizer checkpoint while controlling the training budget provides a useful way to compare delayed consequences of data-mixture choices. Reusing these expensive observations could be valuable for training-system design.
  2. The manuscript is unusually explicit about limitations and negative results. It reports cold-start failure under utility shift, insufficient memory support, inconclusive experiments where constant policies also succeed, and a competitive MLP baseline. These disclosures help distinguish demonstrated behavior from broader aspirations.
Critical weaknesses
  1. The abstract emphasizes 20/20 and 22/22 correct decisions. However, Appendices C.2 and K.1 acknowledge that these experiments lack complete per-query same-anchor exclusion and nearest-neighbor manifests. Freezing predictions before revealing a new branch outcome prevents using that particular outcome in advance, but does not establish that memory excludes historical outcomes from the same checkpoint or closely related states. The authors acknowledge the limitation, but it still affects how much the headline results support the central contribution.
  2. Model configuration requiring clarification. Pages 1, 6, and 9 repeatedly describe a 336M model. However, the shared configuration in section B.2 on page 13 specifies a vocabulary of 32,000, 12 layers, width 768, feedforward width 3,072, and tied input/output embeddings. The core parameter count of a standard dense decoder is approximately 32,000 × 768 + 12 × (4 × 768² + 2 × 768 × 3,072) = 109.5M. Ordinary additional parameters do not explain 336M. This is an internal configuration inconsistency requiring a complete architecture and an actual parameter count.
  3. Practical training value remains preliminary. The experiments primarily assess local action rankings over a fixed horizon. They do not yet show that repeated TIMELOOP decisions improve final training quality or reduce the total computation needed to reach a target quality. The paper does report branch-acquisition costs, and its amortization arithmetic is reasonable under the stated assumptions. However, comparing amortized memory acquisition with exhaustively evaluating every action at every query does not establish an advantage over practical alternatives such as a learned controller or short-horizon evaluation.
Questions for authors
  1. In the shift experiment, are evaluation-query anchors excluded from those acquisitions?
  2. Under identical supervision and evaluation splits, what practically meaningful advantage does explicit retrieval provide over the MLP?
Reviewer_SAyVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

N/A, see Weaknesses

Critical strengths

N/A, see Weaknesses

Critical weaknesses

I highly suspect the submission is fully AI-written. Below I supply some evidence:

  • Sec 3.3, p. 5: "The memory is explicit: a reviewer can inspect the checkpoint identity, query state, action, horizon, endpoint metrics, and utility associated with every retrieved item."
  • Sec 4.1, p. 6: "no new expanded-action rollout is added in this revision, and any completed post-acquisition diagnostic remains separately labelled."
  • Sec 4.2, p. 6: "The full manifests, data-source descriptions, configurations, and metric dictionary are retained in the evidence bundle."
  • Sec 4.4, p. 7: "The local evidence package records the instance identity, source path, manifest, frozen prediction hashes, branch-level outcomes, leakage audits, and the distinction between prospective training and CPU-only post-hoc re-evaluation."
  • Sec 5.2, p. 7: "The export lacks a row-level same-anchor/nearest-neighbor manifest, so this is heterogeneous, protocol-bound evidence rather than a strict unseen-state claim." "Because the export lacks row-level same-anchor exclusions, the result is descriptive mechanism evidence, not a strict leave-one-anchor-out test."
  • Sec 5.4, p. 8: "Because the local package does not retain paired rows for this aggregate controller comparison, we keep it descriptive and do not claim a universal controller gap."
Questions for authors

N/A, see Weaknesses

Reviewer_EqHDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TIMELOOP to address the data mixture selection problem in language model pretraining. The method stores historical training states, data mixtures, future horizons, and their realized future utilities in a Future Memory, and uses nearest-neighbor retrieval to estimate the utility of different mixtures at the current training state. Experiments on a 336M-parameter model with four data domains and five predefined mixture actions show that TIMELOOP selects the mixture with the highest realized utility in all 20 evaluated decision states.

Critical strengths
  1. The paper addresses an important problem in language model pretraining and explores the use of historical long-horizon training outcomes to guide subsequent data mixture decisions.

  2. The proposed method is simple and interpretable, and the paper includes various diagnostic experiments and ablation studies to examine its behavior.

Critical weaknesses

W1. The paper lacks direct comparisons with established data mixture optimization methods. The MLP baseline also achieves comparable performance, leaving TIMELOOP's advantages unclear.

W2. The paper does not provide complete same-anchor exclusion audits for the main 20/20 and 22/22 results, limiting the evidence for generalization to unseen training states.

W3. Experiments are limited to a 336M model and predefined mixture actions. No end-to-end pretraining improvements or compute-matched benefits are demonstrated.

Questions for authors

Q1. What are TIMELOOP's key advantages over existing data mixture optimization methods, and why are direct comparisons with these methods missing?

Q2. Why is the evaluation limited to fixed-horizon loss reduction? Can the authors demonstrate improvements in final pretraining performance or downstream tasks?

Q3. How well does TIMELOOP scale beyond the 336M model and five predefined mixture actions, particularly considering the computational cost of collecting Future Memory?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.