acceptodds
Under review as a conference paper at ICLR 2027

Reading the Tea Leaves, Again: Can Large Language Models Predict Language Model Training Runs?

Abstract

LLM agents now propose research ideas and run the experiments that test them. However, they run every experiment to its end, whereas a researcher babysits a run, reading its configuration and the beginning of the curve, and stops the ones that show no promise. We ask whether LLMs can read a run the same way and build Tea Leaves to measure it, from 682 real training runs across pretraining, fine-tuning, and reinforcement learning (RL). A forecaster sees the first 20% of a run with its configuration, description, and code, and predicts the rest, which requires both curve extrapolation and a working knowledge of training dynamics. Across 11 LLMs and a range of forecasting baselines, the strongest LLMs match or exceed the best curve fitter on smooth loss curves (skill 0.66 against 0.62) and lead on noisy RL reward curves (0.43 against 0.13), but their skill falls as the runs get harder to read: on sparsely logged community runs they lose most of it, and no method reliably flags a failing run in advance. Our analyses identify what makes a run hard to read: its early curve is too noisy or too sparsely logged to extrapolate from. Knowledge of training dynamics helps where the curve misleads; related runs in the prompt cut the error on loss curves by up to 39%, and tools and finer input windows add little for the strongest readers. The benchmark and its database of runs will be released to test LLMs' understanding of training and help research agents decide which runs deserve their compute.

Primary area:
foundation or frontier models, including LLMs
Keywords:
benchmark, training-run forecasting, learning-curve extrapolation, research agents, LLM evaluation, retrieval-augmented forecasting

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_ip94OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Tea Leaves, a benchmark of 682 real language model training runs, to evaluate whether LLMs can forecast future training trajectories from partial curves, configurations, and code. Experiments show that strong LLMs are competitive with specialized forecasters on loss curves and perform better on noisy RL rewards. The authors further investigate observation length, training failure detection, supervised adaptation, historical run retrieval, and agentic tool use, finding that related training histories often provide more value than additional reasoning effort.

Critical strengths
  • The paper identifies an important capability missing from current research agents: anticipating training outcomes before experiments finish. This provides a useful direction beyond automating experiment execution.
  • Tea Leaves covers pretraining, fine-tuning, RL, and community runs, with explicit attention to data provenance and lineage isolation.
  • The contrasting results between loss forecasting and RL forecasting reveal meaningful limitations of current models.
Critical weaknesses
  • My main concerns lie in the gap between forecasting training trajectories and making useful experimental decisions. The paper motivates Tea Leaves through early stopping and compute savings, but its primary evaluation measures pointwise prediction error rather than whether a forecast helps terminate unpromising runs without discarding promising ones. A forecaster can achieve a high skill score while making costly stopping decisions, and accurate trajectory prediction may not even be necessary for effective resource allocation. Although the authors explore failure detection and hyperparameter selection, neither establishes that the proposed forecasting capability improves the efficiency of autonomous experimentation. This weakens the connection between the benchmark results and the paper's broader practical claims.
  • The curated runs come largely from well-instrumented projects, whereas forecasting performance drops sharply on community runs. Although the authors investigate logging sparsity, the results do not fully disentangle the effects of project familiarity, training algorithm, data quality, and distribution shift, leaving generalization to unseen training systems uncertain.
Questions for authors
  • Is there an intrinsic predictability limit for training trajectories?
  • How should an autonomous researcher respond when its forecast is unreliable?
3 comments
AuthorsOpenReview ↗

Thanks for your review! (1/2)

Thank you for recognizing that anticipating training outcomes is an important capability missing from current research agents. We ran four new experiments: two 72-run sweeps and a budget replay (W1), and 129 later runs (W2).

W1: From forecasting to experimental decisions

As you point out, Tea Leaves targets one capability: anticipating training outcomes before experiments finish. This decides our setting: each run is forecast from its first 20% and scored on its own, with a skill score (0 for repeating the last observed value, 1 for a perfect forecast). We agree that showing the link to decisions strengthens the paper. We find that forecasting skill and decision quality are positively related but not identical: the models choose the better of two runs before the curves separate, and on pretraining sweeps they need a third less training budget than keeping the current leader (the run that is ahead at 20%); on fine-tuning and RL sweeps, their skill is not yet sufficient to improve on it.

  • A sweep we trained. To test decisions that the curve cannot support, we further trained 72 new runs on our own (a 124M model; learning rate × cooldown length × a code change; six seeds) in which the cooldown and the code change act only after the 20% cutoff. Asked which of two runs ends lower, GPT-5.6 Sol and Claude Opus 5 are right on all 36 code-change pairs when they read the run descriptions (current leader: 0.46). A second sweep of 72 runs, in which the described change helps in half of the pairs and hurts in the other half, gives the same result (Sol 36 of 36, Opus 35 of 36), where a rule that only notes that a change is described is at 0.50.
  • Does a forecast save compute? To measure it, we replay the benchmark's sweeps: every run is observed for 20% of its budget, and only the runs forecast to end best are continued. On the 10 pretraining sweeps, Sol continues one run per sweep and ends, on average, within 1% of the best final loss (relative to the sweep's spread). Keeping the current leaders needs a quarter or half of the runs, so Sol uses 33.5% and 42.6% less training budget.
  • "A forecaster can achieve a high skill score while making costly stopping decisions." This can happen, and a simulation with forecasters of controlled skill shows why: an error shared by a whole sweep lowers the skill without changing the order of its runs. On Marin, Opus has slightly higher skill than Sol (0.80 vs 0.75) and a larger gap to the best run when keeping one run (0.193 vs 0.010). The skill score stays our primary metric because it is defined for every run and separates models that a compute saving would rate as equal (a saving appears only past a skill level that depends on the sweep); Appendix K reports the run order next to it.
  • "Accurate trajectory prediction may not even be necessary." The same capability can also be queried directly: on the cooldown and code pairs of our sweep, with the run descriptions, asking which run ends lower is more accurate than ordering the same model's forecasts (Sol 0.72 vs 0.54; Opus 0.94 vs 0.62).

Both studies are now in a new Section 4.6, with full tables in a new Appendix K. Additions are in blue; the submitted numbering is unchanged.

(Our response continues in the next comment with W2, Q1 and Q2.)

AuthorsOpenReview ↗

Thanks for your review! (2/2)

W2: Curated and community runs

To separate the factors you list, we ran one analysis for each. Logging quality accounts for most of the gap in the error floor, and the loss/RL contrast reappears inside the community set:

  • Data quality. Thinning the curated curves to community logging density and adding community-level noise raises the lowest error that a smooth hindsight fit can reach (the error floor) from 0.090 to 0.214, against 0.225 on the community runs (Appendix D.7).
  • Training algorithm. Inside the community set, the error relative to repeating the last value is 0.55-0.64 on the 140 fine-tuning and preference-tuning runs and 0.96-1.04 on the 92 RL runs.
  • Project familiarity and distribution shift. On 129 runs created after the submission, and after the models' snapshots, by owners outside the benchmark (mostly pretraining losses), the LLMs' error is 0.73-0.74 times that of LC-PFN, the best curve fitter of the paper.

This breakdown is in the new Appendices L.5 and L.6, linked from Appendix D.7.

Q1: Is there an intrinsic predictability limit?

We believe there is, and that it depends on what the forecaster can read. Part of it is randomness that no forecaster can remove: in our sweep, runs that differ only in the random seed already end 0.006 apart in final loss. The rest is information: what happens after the cutoff is often not determined by the curve seen so far.

This second part can move. As discussed in Section 5 of the draft, a better signal may lie in the data, outputs, gradients, and checkpoints that a researcher inspects next, and our agent mode (Section 4.5), where the model opens the run's repository and runs code in a sandbox, is the place to provide them. Current models do not yet turn this access into better forecasts, which motivates agentic post-training as future work: training models to inspect a run as a researcher would. There is room to gain: a curve fitted in hindsight reaches a skill of 0.90 on the 129 new runs, against 0.76 for the best forecaster.

Q2: What should an agent do when its forecast is unreliable?

In our view, a forecast is a signal for a decision, not the decision itself, so the first requirement is that the agent knows how much to trust it. This is why the benchmark asks for quantiles and scores them: the 80% intervals of GPT-5.6 Sol contain the final value 80% of the time, so a wide interval is a usable warning. When the warning fires, the agent should prefer the cheap and reversible action over the irreversible one: keep the run going and look again later, because waiting costs a bounded amount of compute while stopping a good run loses it, or fall back to the simple rule of keeping the runs that are currently ahead. In our 72-run sweep, the leading runs at 20% are closer than the seed noise, and training all runs to 50% before choosing finds the best run, at 62.5% of the full budget instead of 40%. More broadly, we see these forecasts today as signals that assist a human researcher, and an autonomous agent should hand the decision back when its forecast is unreliable.

Both points are now in Section 5.

Thank you for your time and effort in reviewing our paper. We have incorporated your suggested experiments and believe they have strengthened the paper. We hope you could consider increasing your rating in light of these revisions.

AuthorsOpenReview ↗

Thanks for your review!

Dear Reviewer ip94,

Thank you again for your constructive review. As the discussion period is ending soon, we would greatly appreciate your feedback on our response. Following your comments, we trained two controlled sweeps and replayed the benchmark's sweeps to connect forecasting to decisions (new Section 4.6 and Appendix K), analyzed 129 later runs and the community runs to separate the factors behind the gap (Appendices L.1, L.5, and L.6), and added our view on the predictability limit and on unreliable forecasts to Section 5.

We are happy to answer any further questions. We are grateful for your positive assessment of the paper; if our response has addressed your remaining concerns, we would appreciate your continued support and any update to your score that you find appropriate.

Best regards, Authors

Reviewer_JQpUOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TEA LEAVES, a benchmark of 682 publicly sourced language-model training runs spanning pretraining, fine-tuning, reinforcement learning (RL), and community experiments. Given the initial 20% of a target metric curve, together with the numerical configuration and contextual information such as task descriptions and code excerpts, a forecaster predicts subsequent points and uncertainty quantiles. The main metric is a geometrically aggregated error ratio relative to repeating the last observed value. The paper evaluates frontier and open-weight LLMs, conventional curve fitters, time-series foundation models, and models trained on historical runs. Strong LLMs are competitive on smooth loss curves and show a clearer advantage on noisy RL reward curves. Predictive skill degrades on sparsely logged community runs, and early detection of failing runs remains difficult. The paper additionally studies observation density, fine-tuned TeaReader models, retrieval from past training runs, and tool-augmented forecasting.

Critical strengths
  1. Valuable benchmark construction and breadth of empirical analysis. The compilation of real training trajectories across distinct learning regimes, including a separate in-the-wild setting, addresses a useful gap between synthetic time-series forecasting and practical ML experimentation. The attempt to track run lineage, control sweep overlap, and provide detailed protocols and paired statistical analyses makes this a potentially useful community resource (Sections 3-4; Appendices A-C).
  2. Several informative and appropriately nuanced findings. The strong RL forecasting results relative to LC-PFN are notable (e.g., Claude Opus 5: paired Delta S = 0.29, 95% CI [0.18, 0.41]; Table 7). The analyses also identify limitations rather than reporting only wins: the model performs less well on sparse community logs, early failure warnings are unreliable, and retrieving related historical runs substantially reduces loss-forecasting error (Section 4.4; Table 9). These observations are interesting independently of the broader application claim.
Critical weaknesses

1. The connection between forecasting improvements and useful training decisions remains unestablished. The paper motivates forecasting through early stopping and compute allocation for research agents, but its primary task measures prediction error at future points. More accurate trajectories do not necessarily yield better decisions about which runs to retain, stop, or prioritize. The online failure-monitoring experiment is relevant, but LLMs do not outperform boosted trees on curve statistics; the recommendation experiment in Appendix H is also a limited offline evaluation. The evidence therefore establishes a training-curve forecasting benchmark more clearly than a practical improvement to research efficiency.

2. The TeaReader results lack a fully independent final evaluation. Appendix G and Table 26 explicitly state that the reference recipe was selected after inspecting the frozen, RL, and community test scores. Subsequent experiments fix that recipe, but inherit its test-informed selection and do not independently confirm it. Moreover, the 2,641- and 3,002-run pools contain 22 records subsequently found to share sweep-family relations with test runs, affecting the intermediate points in Figure 4. The disclosure is appreciated, but the affected experiments should be rerun after cleaning, and the main TeaReader conclusions should be validated on an untouched test set. This concern specifically affects the trained-reader study rather than invalidating all zero-shot benchmark results.

3. The interpretation in terms of understanding training dynamics exceeds what the experiments establish. Appendix E.1 shows that replacing a run’s semantics with those of another run from the same project leaves forecasting error largely unchanged, and a task description alone recovers the semantic benefit for one model. These findings suggest that coarse task or project priors may explain part of the gain. Likewise, strong pairwise setting–curve matching deteriorates toward chance under four-way matching, while matching derived from actual forecasts is near chance. The evidence does not sufficiently distinguish run-specific understanding from task identification and relative comparison between candidates.

4. Some baselines are constrained by input adaptation or training-data coverage. Appendix A.2 notes that TimesFM receives only ten points within one input patch, while FT-PFN’s configuration slots are largely filled with neutral values, making it effectively a curve-only predictor. After test-sweep exclusion, some trained baselines have very few relevant training runs. The retrieval comparison also lacks a non-LLM baseline using the same configuration-based neighbors as the LLM. Competitive results from LC-PFN make the evaluation useful, but better-matched baselines are needed to support broad claims of superiority over specialized predictors.

Questions for authors
  1. Can the authors replay complete historical sweeps under an explicit stopping or budget-allocation policy, reporting compute saved, promising runs stopped incorrectly, and degradation in the best final outcome? Do better forecasts produce better decisions?

  2. Can the affected TeaReader experiments be rerun after removing the 22 related records, followed by evaluation on new projects or later runs that were not used in recipe development?

  3. Given the limited effect of within-project semantic substitution, what demonstrates the use of run-specific training knowledge? Could controlled changes to configurations, learning-rate schedules, or code test whether forecasts respond in the correct direction?

  4. Can the authors include stronger information-matched baselines, such as curve transfer using the same configuration-retrieved neighbors, time-series models with appropriate input lengths, and configuration-conditioned predictors with sufficient relevant training data?

3 comments
AuthorsOpenReview ↗

Thanks for your review! (1/2)

Thank you for the detailed review, and for noting that the observations are interesting independently of the application claim. The authors sincerely appreciate your suggestions. We ran all four experiments you asked for: a budget replay of complete sweeps (Q1), TeaReader retrained on cleaned pools and tested on 129 later runs from new owners (Q2), 72 newly trained runs with controlled changes (Q3), and baselines given the same retrieved neighbours (Q4).

W1 and Q1: Do better forecasts produce better decisions?

Tea Leaves benchmarks a model capability, anticipating a run's outcome before it finishes, and decisions are where this capability is used. We find that better forecasts tend to give better decisions: forecasting skill (0 for repeating the last observed value, 1 for a perfect forecast) and decision quality are positively related but not identical, and what matters most is whether the forecast orders the runs correctly. On pretraining sweeps the forecasts need a third less training budget than keeping the current leader (the run that is ahead at 20%); on fine-tuning and RL sweeps, the models' skill is not yet sufficient to improve on it.

The replay you suggested. To measure this, we replay the benchmark's sweeps under an explicit budget policy: all n runs of a sweep are trained to 20%, and only the m runs with the best predicted final value are completed (m ∈ {1, ⌈n/4⌉, ⌈n/2⌉}; budget 0.2 + 0.8·m/n of training everything). The gap to the best run is measured relative to the spread of final losses in the sweep. As the table shows, on the two pretraining sources with sweeps (Marin and DCLM, Table 1), GPT-5.6 Sol reaches the best run's level with one run per sweep, where keeping the current leaders needs a quarter to a half of the runs:

SweepsPolicy (skill)Runs kept per sweepBudget
Marin pretraining (6 sweeps, 63 runs)GPT-5.6 Sol (0.75)10.293
SameKeep the current leaders⌈n/4⌉0.441
DCLM pretraining (4 sweeps, 22 runs)GPT-5.6 Sol (0.71)10.362
SameKeep the current leaders⌈n/2⌉0.631
  • Compute saved: at a gap of at most 0.01, Sol needs 33.5% and 42.6% less budget than keeping the current leaders (about two thirds less than training every run). On the fine-tuning and RL sweeps, the forecasts do not improve on keeping the current leaders.
  • Degradation in the best final outcome: this is the gap to the best run, and it is at most 1% of the sweep's spread in every row of the table (0.010 for Sol on Marin, 0.000 in the other three rows).
  • Promising runs stopped incorrectly: with a quarter of the Marin runs kept, 8% of the near-best runs (within 5% of the sweep's spread from the best) are stopped under the current-leader policy (more runs are near-best than can be kept), 14% under Sol and 31% under Claude Opus 5.

Why the two are not identical. Across the LLM forecasters, higher skill goes with a smaller gap (pooled Spearman 0.23 [0.08, 0.49]), but the gap follows most closely whether the runs are ordered correctly: on Marin, Opus has slightly higher skill than Sol (0.80 vs 0.75) and a larger gap when keeping one run (0.193 vs 0.010). A simulation with forecasters of controlled skill gives the reason: an error shared by a whole sweep lowers the skill without changing the order of its runs.

Why the benchmark scores skill. The saving is a thresholded function of skill: in the simulation, a forecaster saves compute only once its skill passes a level that depends on the sweep (0.6 to 0.95 on Marin). A benchmark scored by saving would rate most current models as equal, while the skill score separates them and tracks progress toward that level.

A sweep we trained. To test decisions that the curve cannot support, we ask which of two runs ends lower on the 72 new runs described under Q3. With the run descriptions, both LLMs beat the current leader in all six seeds (p = 0.031). Training new runs is costly for a pure benchmark setting, but it gives direct evidence connecting the skill to the decision.

Failure monitoring (W1). On failure monitoring, the capability is emergent as well: without any training for the task, the LLMs rank failing runs as well as boosted trees fitted on curve statistics (AUROC 0.67 vs 0.65 at a lead of 10% of the budget; Appendix F), where the specialized curve predictors are at chance (0.43-0.48). No monitor is reliable yet (the best catches 0.41 of the failures online), and we see this as one of the capabilities that the benchmark can help future work improve.

Both studies and the simulation are now in a new Section 4.6, "From forecasts to decisions", with the protocol and full tables in a new Appendix K. All additions are marked in blue in the revised version, and the section and table numbers of the submitted version are unchanged.

(Our response continues in the next comment with W2/Q2, W3/Q3, and W4/Q4.)

AuthorsOpenReview ↗

Thanks for your review! (2/2)

W2 and Q2: Independent evaluation of TeaReader

To give TeaReader an independent evaluation, we ran both checks you asked for. The conclusion of Section 4.3 holds on an untouched test set and after cleaning the pools.

  • Untouched test. We collected 129 runs created after the submission date by owners that appear in no test set and not in the reader's training pool (110 pretraining, 19 RL), and ran the submitted adapters, prompts, and parser unchanged. The 2B reader scores 0.724, against 0.736 / 0.742 for GPT-5.6 Sol / Claude Opus 5 and 0.644 for LC-PFN.
  • Cleaned reruns. We removed the 22 records from the 2,641- and 3,002-run pools and retrained the 2B readers with the original recipe and three seeds. Skill on the benchmark's test sets changes by at most 0.022, and the gain on held-out sources over the 918-run pool of Section 4.3 remains (0.779-0.806 vs 0.680).

As in Section 4.3, the trainability result is for loss curves. Both checks are now reported in Section 4.3, with details in the new Appendices L.1 and L.7.

W3 and Q3: Run-specific understanding or task prior?

Appendix E.1 shows that the semantic gain is carried by information shared within a project. We also agree with your reading of the matching results: the models compare two candidates rather than evaluate one (four-way accuracy is at most 0.34, and matching derived from forecasts is at chance; Appendix E). To test whether the forecasts also use run-specific information, we ran the controlled test you propose. We find that they do respond to a run-specific change, in the correct direction and with a predicted effect close to the measured one, when the description states the change.

We trained 72 new runs (a 124M model on FineWeb; three learning rates × two cooldown lengths × with or without a code change that freezes the embeddings; six seeds). The cooldown and the code change act only after the 20% cutoff, so the observed curve does not show them.

  • Code. The change raises the final loss by +0.116 nats. In its forecasts, Sol predicts +0.001 from the configuration alone and +0.098 once the description states the change; describing a change that does nothing leaves the forecast unchanged within noise (-0.035 [-0.087, +0.017]). Asked directly which of two runs ends lower, both LLMs are right on all 36 code pairs, from the descriptions and also from the two executed scripts alone.
  • Code, both directions. In this sweep the described change always hurts, so a rule that only notes that a change is described would also be right. To rule this out, we trained a second sweep of 72 runs in which the same change goes both ways: freezing the embeddings at step 150 raises the final loss, and unfreezing them at step 150, in runs where they were frozen from step 50, lowers it. The two runs of a pair have the same curve at the 20% cutoff. With the descriptions, Sol orders 36 of 36 pairs correctly and Opus 35 of 36; the keyword rule is at 0.50, the current leader at 0.54, and both LLMs are at chance without the descriptions.
  • Learning-rate schedule. Opus orders 77% of the cooldown pairs correctly from the curve and the configuration (current leader: 45%); Sol does not, since it expects a longer cooldown to hurt.
  • Learning rate. Its effect is already visible in the curve at the cutoff, and every method orders these pairs correctly.

This test is in the new Section 4.6 and Appendix K (the second sweep in Appendix K.7), and Appendix E.1 now points to it.

W4 and Q4: Information-matched baselines

To match the information across methods, we added the baselines you listed. With the same related runs, a specialized predictor becomes competitive with the LLM, and the two are complementary, which supports the finding of Section 4.4 that related runs are the most valuable context.

  • Same retrieved neighbours. We gave FT-PFN (the configuration-conditioned curve predictor among our baselines, Appendix A.2) the same three related runs that the LLM sees, with an explicit configuration encoding. It scores 0.820 against 0.799 for Sol with retrieval: Sol is better on pretraining and FT-PFN on fine-tuning. A direct curve transfer from the same neighbours scores 0.597.
  • Appropriate input lengths. Appendix F.3 (Table 23) gives the time-series models denser windows: Chronos-2 improves (error ratio 0.72) and TimesFM does not (1.73).
  • Enough relevant training data. Appendix D.8 (Table 17) retrains NCPL (a predictor trained on logged runs) with the test run's sweep in training (0.79) and held out (0.28). Trained predictors are strong when similar runs are in their training set; the LLMs need none from the target project.

The same-neighbour baselines are in the new Appendix L.2, with a summary in Section 4.4.

Thank you for your time and effort in reviewing our paper. We have incorporated your suggested experiments and believe they have strengthened the paper. We hope you could consider increasing your rating in light of these revisions.

AuthorsOpenReview ↗

Thanks for your review!

Dear Reviewer JQpU,

Thank you again for your detailed review. As the discussion period is ending soon, we would greatly appreciate your feedback on our response. We ran all four experiments you asked for: the replay of complete sweeps under an explicit budget policy (new Section 4.6 and Appendix K), TeaReader retrained on cleaned pools and tested on 129 later runs from new owners (Appendices L.1 and L.7), two newly trained sweeps with controlled changes (Appendix K), and baselines given the same retrieved neighbours (Appendix L.2).

We are happy to answer any further questions. If our response has addressed your concerns, we would be grateful if you could consider raising your score.

Best regards, Authors

Reviewer_yQA3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a benchmark called Tea Leaves that measure the forecasting accuracy for LLM training. It asks a forecaster to predict the remaining metrics based on first 20% of the training metrics, and the hope is that if we can find a good forecaster, we can use it to save cost by selecting more promising runs / stopping less promising ones. The benchmark contains 3 types of data: pre-training, fine-tuning and RL, with both well-curated data as well as community-collected data from HuggingFace. The paper studies a few previous methods on this benchmark as well as a few novel approaches proposed, including a fine-tuned model on this task called TeaReader. The paper also studies if rich textual data (like the training setup, task info, etc.) are useful when the forecaster is a LLM, but mixed results were found. Best LLMs are as good as curve-fitting baselines but seems to be superior on RL tasks. All approaches seem to have a worse performance on the community-collected data.

Critical strengths
  • The benchmark and data are both useful to the LLM community. It covers the most important 3 types of LLM training as well as difference data sources.
  • The analysis is comprehensive with different questions studied and a wide range of baselines covered. Results indicate the current status of how well different methods perform on this task and could be useful for both practitioners doing these tasks or researchers would want to improve this benchmark.
Critical weaknesses
  • The main metric, skill score, does not measure the actual decision power or how much a forecaster would save the training cost. It's unclear how a particular score mean in the actual cost saving, and I don't think a score would properly reflect it, simply because what also matters are metrics like how calibrated the forecaster is. If the claim is really about benchmarking this, the benchmark should have be defined as some measure of cost saving from simulated training runs using the historical data, similar to what section 4.4 or the sweep recommendation experiments in the appendix.
  • The usefulness of the extra experiment information remains unclear.
Questions for authors
  • If we fix the observed curve and numerical configuration, do the experiment description and code each provide a consistent additional benefit?
  • Why there are cases where the full context make forecasts worse for certain models or tasks?
  • How would someone map the level of skill score to the actual cost saving? Is skill score enough or a simulated experiment would be more informative (re. the point above in weaknesses).
3 comments
AuthorsOpenReview ↗

Thanks for your review! (1/2)

Thank you for recognizing the benchmark's usefulness and the breadth of the analysis. The authors really appreciate your constructive feedback. We ran the experiments you suggested and address your comments below.

W1 and Q3: What we benchmark, and how skill relates to cost savings

Tea Leaves benchmarks a model's capability: anticipating how a training run will end from its first 20%. We score it with a skill score (0 for repeating the last observed value, 1 for a perfect forecast) because it is defined for every run with no sweep needed, so that we can include the 232 community runs.

Cost saving is what this capability is for (Section 5) in the downstream. We find that skill and cost saving are positively related but not identical: on pretraining sweeps the forecasts need a third less training budget than keeping the current leader (the run that is ahead at 20%), while on fine-tuning and RL sweeps the models' skill is not yet sufficient to improve on it. The skill score stays our primary metric because it is defined on every run; the revised Section 4.6 reports the cost simulation next to it.

A second reason is resolution: the saving is a thresholded function of skill. In our simulation, a forecaster saves nothing, or needs more budget than keeping the current leaders, until its skill passes a level that depends on the sweep (0.6 to 0.95 on the Marin sweeps), and on the fine-tuning and RL sweeps no current model saves anything. A benchmark scored by saving would rate most current models as equal, while the skill score separates them and tracks progress toward that level. Additions are in blue; the submitted numbering is unchanged.

Calibration is part of the benchmark: forecasters output 10%, 50%, and 90% quantiles (Section 3.1), Appendix C scores them with a quantile loss, and GPT-5.6 Sol's 80% intervals contain the final value 80% of the time.

The simulation on historical data you suggested. To measure the cost saved, we replay the benchmark's sweeps: every run is observed for 20% of its budget, and the m runs forecast to end best are continued (budget 0.2 + 0.8·m/n of training all n runs). As the table shows, on the two pretraining sources with sweeps (Marin and DCLM, Table 1), Sol needs one run per sweep to end, on average, within 1% of the best final loss (relative to the sweep's spread), with 33.5% and 42.6% less budget than keeping the current leaders (about two thirds less than training every run):

SweepsPolicy (skill)Runs kept per sweepBudget to end within 1% of the best
Marin pretraining, 6 sweepsGPT-5.6 Sol (0.75)10.293
SameKeep the current leaders⌈n/4⌉0.441
DCLM pretraining, 4 sweepsGPT-5.6 Sol (0.71)10.362
SameKeep the current leaders⌈n/2⌉0.631

A sweep we trained ourselves. To test decisions that the curve cannot support, we trained 72 runs (a 124M model; learning rate × cooldown length × a code change; six seeds) in which the cooldown and the code change act only after the cutoff. Asked which of two runs ends lower, Sol and Claude Opus 5 are right on all 36 code-change pairs when they read the run descriptions (current leader: 0.46). A second sweep of 72 runs, in which the described change helps in half of the pairs and hurts in the other half, gives the same result (Sol 36 of 36, Opus 35 of 36), where a rule that only notes that a change is described is at 0.50. This is costly for a pure benchmark, but it is direct evidence connecting the skill to the decision.

From skill to saving (Q3). To map one to the other, we simulated forecasters of controlled skill on the same sweeps. If a forecaster's errors are shared within a sweep, as the last value's mostly are, a skill of 0.6 suffices to keep one run on Marin; if they are independent across runs, it needs about 0.95, because a shared error lowers skill without changing the order of the runs. So a skill level maps to a saving only together with the sweep. All of this is in a new Section 4.6 and Appendix K.

(Our response continues in the next comment with W2/Q1 and Q2.)

AuthorsOpenReview ↗

Thanks for your review! (2/2)

W2 and Q1: What description and code add

The benefit depends on how specific the text is to the run. In the draft, the semantics raise Claude Opus 5's skill on RL from 0.33 to 0.43 (Table 10). To separate the two components as you asked, we fix the curve and the numeric configuration (C) on 200 questions (100 pretraining, 100 fine-tuning; only 24 of the 150 RL runs carry code) and add the description (D) or the code excerpt (K). As shown below, neither adds a consistent benefit on these logs (error ratios; < 1 means the added part helps):

AddedGPT-5.6 SolClaude Opus 5
Description (CD vs C)0.96 [0.89, 1.05]1.01 [0.91, 1.12]
Code (CK vs C)1.00 [0.91, 1.10]1.00 [0.89, 1.12]

A likely reason is that the text is rarely specific to a run: the 150 pretraining runs share most parts of description texts. When the description does state a run-specific fact, it helps: in our 72-run sweep, describing the code change moves Sol's predicted effect from +0.001 to +0.098 nats (measured: +0.116). This study is the new Appendix L.3, linked from Section 4.1.

Q2: Why full context sometimes hurts

It hurts when the model over-applies a general expectation that it takes from the context. In the draft, full context lowers Claude Opus 5's fine-tuning skill by 0.19 and raises GPT-5.6 Sol's by 0.06 (Appendix C). To find out why, we went back to the individual answers. Three-quarters of Opus's drop comes from the 50 Open-R1 runs, which train for several epochs. Once told the number of epochs, Opus expects memorization and predicts 2.7 times the loss decline that actually happens (0.67 times from the curve alone); 43 of its 47 explanations mention memorization or overfitting. Sol reads the same field and adjusts by the right amount (from 0.51 to 0.66 times the real decline), which is where its gain comes from.

One run as an example (a 10-epoch Open-R1 run; the loss is 0.625 at the 20% cutoff and ends at 0.526):

Final loss predictedCurve onlyFull context
Claude Opus 50.5350.210
GPT-5.6 Sol0.5890.529

So the same piece of context moves both models in the same direction, and what differs is how much. This analysis is the new Appendix L.8.

Thank you for your time and effort in reviewing our paper. We have incorporated your suggested experiments and believe they have strengthened the paper. We hope you could consider increasing your rating in light of these revisions.

AuthorsOpenReview ↗

Thanks for your review!

Dear Reviewer yQA3,

Thank you again for your constructive review. As the discussion period is ending soon, we would greatly appreciate your feedback on our response. Following your suggestions, we added the cost simulation on historical sweeps and a mapping from skill to cost savings (new Section 4.6 and Appendix K), the effect of the description and code with the curve and configuration fixed (Appendix L.3), and an analysis of why the full context sometimes hurts (Appendix L.8).

We are happy to answer any further questions. If our response has addressed your concerns, we would be grateful if you could consider raising your score.

Best regards, Authors

Reviewer_o98uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Recent progress toward self-training, where models take part in their own pretraining, raises the question of how well LLMs can manage experiments. LLMs have been shown to generate and evaluate research ideas, but their ability to manage running experiments is less clear. Human researchers can quickly interpret an experiment’s progress and stop training that is not beneficial; it is unclear whether LLMs can do the same. Motivated by this, the authors propose predicting the learning trajectory of a model training experiment. Given the experimental setup and details plus the first 20% of the learning trajectory (e.g., the loss curve), the model must predict the rest of the trajectory. The authors release a dataset of 682 real experiments and evaluate LLMs in zero-shot and fine-tuned settings, using MAE as the evaluation metric.

Critical strengths
  • The paper offers useful insight into whether LLMs can reason about and anticipate training dynamics, which is relevant to the broader goal of automating research workflows.
  • The authors release a dataset of 682 real experiments, which may be useful to the community.
  • The experimental evaluation, covering both zero-shot and fine-tuning settings with MAE as the metric, appears sound. I did not find issues with the technical soundness of the experiments.
Critical weaknesses
  • There is a limitation in novelty. The task closely resembles existing work on performance and learning-curve prediction. The paper does not clearly explain what is fundamentally new beyond framing the problem with LLMs as predictors.
  • Disconnect between motivation and task. The paper is motivated by enabling machines to manage experiments autonomously, but the actual task is only trajectory prediction. Managing experiments involves decisions, such as whether to stop, continue, or change a run, and the paper does not evaluate these.
  • In practice, researchers rarely judge a single run alone. They compare runs and decide whether experiment A is better than experiment B. A comparative formulation, predicting which of two partially observed runs will perform better, would be harder but more realistic and better aligned with the stated motivation.
  • Recommending the next experiment based on current results would be a more practical and compelling step toward autonomous experiment management. The authors could at least discuss how their setup relates to or could support this.
  • The release of 682 experiments is appreciated, but the dataset’s scale and diversity seem modest. The paper does not convincingly show that it captures enough variety to support general conclusions.
  • Insufficient description of the data. The task and data distribution are unclear. It is not stated which kinds of tasks the experiments cover (e.g., NLP, classification, clustering, language modelling), or what the training data distribution looks like for each case. Without this, it is hard to judge how well the results generalize.
Questions for authors

Please refer to weaknesses

3 comments
AuthorsOpenReview ↗

Thanks for your review! (1/2)

Thank you for recognizing the soundness of the evaluation and the insight into whether LLMs can anticipate training dynamics. The authors sincerely appreciate your constructive comments and suggestions!

W1: Our contribution and the connection between the motivation and the task

Learning-curve prediction is the closest line of work; we discuss it in Section 2 and compare with its strongest methods (Section 3.3). To the best of our knowledge, we are the first to show (and systematically analyze) that a general-purpose LLM has the emergent capability to forecast a training run, with comparable performance to specific curve-prediction models (Sections 1 and 4.1: skill (1 minus the error relative to repeating the last value) 0.66 vs 0.62 on loss curves, and 0.43 vs 0.13 on RL runs). Moreover, in Sections 4.3-4.4, we show how this capability can be improved with extended training or retrieval.

Besides, unlike the existing methods we cover exhaustively in our experiments, LLMs have the potential to utilize the information from the semantics of the experiments; e.g., in Table 10, proprietary models like Claude Opus 5 and GPT-5.6 Sol show better prediction on RL runs with the full semantics (Opus: 0.33 to 0.43). As agents, they can also open a run's repository and run code (Section 4.5), and could inspect its data and outputs, which a curve predictor cannot. We hope the introduction of these settings would inspire future research on what contexts/training help LLMs predict the learning curves, as well as how to leverage this capability in downstream applications.

W2-4: Our motivation, benchmarking LLM capability, and what it opens

Validating and benchmarking the LLM capabilities in predicting the curves would be the first step. At this stage, the authors identify the capabilities of "reading the tea leaves as human researchers" as an observation that motivates the following directions, as mentioned in Section 5 of the draft: (1) how to add existing training runs to the data mixture for model trainers; (2) studying what contexts can help the model predict and how; (3) using the forecasts inside a research loop.

Our current goal is still asking the models to provide signals to assist human researchers in making decisions. Yet, the authors sincerely appreciate and agree with your idea of pointing out the extension of capability to realistic scenarios. An important follow-up work can be: autonomous experiment management. Following your suggestions, we ran a pilot experiment for each decision you list. In short, the capability we benchmark carries over to these decisions: the models pick the better of two runs before the curves separate, and on pretraining sweeps they need a third less training budget than keeping the current leader (the run that is ahead at 20%), while on fine-tuning and RL sweeps their skill is not yet sufficient to improve on it.

  • Compare A with B, and change a run (W3, W2): a sweep we trained. To test decisions that the observed curve cannot support, we trained 72 new runs (a 124M model on FineWeb; three learning rates × two cooldown lengths × with or without a code change; six seeds). The cooldown and the code change only take effect after the 20% cutoff. We then ask which of two runs will end lower. Once the LLMs read each run's configuration and description, they pick the better run in every code-change pair, where keeping the current leader is at chance:
MethodLearning-rate pairs (60)Cooldown pairs (33)Code-change pairs (36)
Keep the current leader1.000.450.46
GPT-5.6 Sol1.000.451.00
Claude Opus 51.000.731.00

Both LLMs beat the current leader in all six seeds (p = 0.031). Training new runs is costly for a pure benchmark setting, which is why Tea Leaves is built on logged runs, but it gives direct evidence connecting the skill to the decision. On the benchmark's own sweeps, the same question improves on the current leader in 7 (Sol) and 10 (Opus) of the 21 cases where the leader is not the final best, and switches away from a leader that was the final best in 4 of the other 51.

  • Stop or continue (W2): a replay of the benchmark's sweeps. To measure the compute saved, every run of a sweep is observed for 20% of its budget and only the runs forecast to end best are continued. On the pretraining sweeps (6 Marin and 4 DCLM sweeps; 85 runs), Sol continues one run per sweep and ends, on average, within 1% of the best final loss (relative to the sweep's spread). Keeping the current leaders needs a quarter (Marin) or half (DCLM) of the runs for the same result, so Sol uses 33.5% and 42.6% less training budget. On the fine-tuning and RL sweeps, the forecasts do not improve on keeping the current leaders. Skill and saving are positively related but not identical.

This discussion is the new Section 4.6, "From forecasts to decisions", with full tables in a new Appendix K.

(Continued in the next comment with W4-6.)

AuthorsOpenReview ↗

Thanks for your review! (2/2)

W4 (continued): Recommending the next experiment, and autonomous experiment management

Our draft takes the first steps in this direction. Section 4.4 and Appendix H build a database of 1,726 past runs that the model queries: on 70 questions about which setting of a sweep ends best, GPT-5.6 Sol with retrieved past sweeps picks the best one 61% of the time, against 39% at random. Section 4.5 and Appendix I give the model an agent harness (reading files, running code, searching literature and code).

A benchmark for autonomous experiment management would close the loop around these pieces. We see it as follows:

  • Environment: a sweep under a fixed compute budget, either replayed from logged runs (cheap and reproducible, as in our replay) or trained live (as in our 72-run sweep).
  • Tools: the harness of Appendix I and the run database of Section 4.4, plus actions to launch, stop, continue, or modify a run.
  • Metrics: the best final result under the budget, the compute saved at equal quality, and the promising runs stopped by mistake.
  • Settings: what the agent can see (curves, configurations, code, and descriptions, as in our information conditions) and how early it has to decide.

In the current paper, we focus on the emergent capability that such an agent needs first, anticipating the outcome of a run, and we believe this benchmark is an important future work motivated by our observations. We have added this discussion to Section 5 ("Toward autonomous experiment management").

W5-6: Our dataset: scales and diversity

Regarding dataset scale, the included 682 experiment runs each have eight future points as forecasting questions to predict on the main grid. These runs form our dataset after our strict filtering of 17,805 collected run records (Section 3.2 and Table 5 in Appendix B): a run is kept only if its target metric and planned budget are defined, its configuration and code or description are visible, and its curve is complete around the 20% mark; runs that resume or branch from one another stay on the same side of the split, and the test sets were frozen before any model was evaluated. Compute is only partly logged: the 100 Marin pretraining runs alone used about 7,300 accelerator-hours (4,100 GPU-hours and 3,200 TPU-chip-hours).

To check that the conclusions hold beyond these runs, we also evaluated all frozen systems on 129 runs created after the submission by new owners: the LLMs keep their advantage over LC-PFN, the paper's best curve fitter (skill 0.74 against 0.64). We will release the time-cut-off data together with our dataset as well.

As for the diversity of our data, we show the sources of the experimental runs in Table 1, e.g., more than twenty repositories for the RL runs. More details are in Appendix B in the original draft. All runs train language models, so the categories are training objectives rather than downstream tasks such as classification or clustering. The detailed tasks and statistics are as follows:

SetRunsSourcesModels and scaleTask types (examples)Budget
Pretraining150Marin, DCLM, OLMoEMoE and dense decoders, 1-48 layersnext-token prediction (C4, RefinedWeb, RedPajama where named)0.05-130B tokens
Fine-tuning150Marin, Open-R121M-1.4B and 3B-35Binstruction tuning with the Tülu 3 recipe (100); SFT in the Open-R1 project (50)3,834 steps or 3-15 epochs
RL15024 projectsGPT-2, Qwen, Llama, OLMo, Gemma; 0.12-30Bmath 30 (e.g., GRPO on math prompts), RLHF with a learned reward model 30, reasoning puzzles 10, code 9, agentic and tool use 8, games 3, other 4, unrecorded 5648-15,492 steps
In the wild232Hugging Face Hub, 184 authorsQwen, Mistral, Llama, Gemma; 0.1-70BSFT 85 and preference tuning 55 (e.g., MATH, GSM8K, UltraFeedback, medical QA, translation); RL 9267-149,068 steps

Table 1 gives one example question per set, Appendix B (Figure 7) shows one run of each as logged, and Appendix D.6 (Table 16) reports the RL results by project. These statistics and the 129 later runs are the new Appendix L, in blue, linked from Section 3.2 and Appendix B.

The data that the runs train on is in the table above (e.g., C4 and RefinedWeb for pretraining, the Tülu 3 mixture for fine-tuning, math and code tasks for RL). For the training data of our trained reader (TeaReader), we introduce our filtering criterion in Section 3.2 and Appendix B (Table 5): we remove every test run, every run that shares a lineage or a sweep with a test run, and runs with incomplete curves. The training data distribution is 918 runs (7,724 examples): 218 pretraining, 52 synthetic-task, 280 fine-tuning or preference-tuning, and 368 RL runs (Appendix G.1).

Thank you for your time and effort in reviewing our paper. We have incorporated your suggested experiments and believe they have strengthened the paper. We hope you could consider increasing your rating in light of these revisions.

AuthorsOpenReview ↗

Thanks for your review!

Dear Reviewer o98u,

Thank you again for your constructive review. As the discussion period is ending soon, we would greatly appreciate your feedback on our response. Following your suggestions, we ran a pilot study for each decision you listed (comparing two runs, changing a run, and stopping or continuing a run; new Section 4.6 and Appendix K), discussed a benchmark for autonomous experiment management (Section 5), and added the task and data statistics you asked for (Appendix L).

We are happy to answer any further questions. If our response has addressed your concerns, we would be grateful if you could consider raising your score.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.