- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Tea Leaves, a benchmark of 682 real language model training runs, to evaluate whether LLMs can forecast future training trajectories from partial curves, configurations, and code. Experiments show that strong LLMs are competitive with specialized forecasters on loss curves and perform better on noisy RL rewards. The authors further investigate observation length, training failure detection, supervised adaptation, historical run retrieval, and agentic tool use, finding that related training histories often provide more value than additional reasoning effort.
Critical strengths
- The paper identifies an important capability missing from current research agents: anticipating training outcomes before experiments finish. This provides a useful direction beyond automating experiment execution.
- Tea Leaves covers pretraining, fine-tuning, RL, and community runs, with explicit attention to data provenance and lineage isolation.
- The contrasting results between loss forecasting and RL forecasting reveal meaningful limitations of current models.
Critical weaknesses
- My main concerns lie in the gap between forecasting training trajectories and making useful experimental decisions. The paper motivates Tea Leaves through early stopping and compute savings, but its primary evaluation measures pointwise prediction error rather than whether a forecast helps terminate unpromising runs without discarding promising ones. A forecaster can achieve a high skill score while making costly stopping decisions, and accurate trajectory prediction may not even be necessary for effective resource allocation. Although the authors explore failure detection and hyperparameter selection, neither establishes that the proposed forecasting capability improves the efficiency of autonomous experimentation. This weakens the connection between the benchmark results and the paper's broader practical claims.
- The curated runs come largely from well-instrumented projects, whereas forecasting performance drops sharply on community runs. Although the authors investigate logging sparsity, the results do not fully disentangle the effects of project familiarity, training algorithm, data quality, and distribution shift, leaving generalization to unseen training systems uncertain.
Questions for authors
- Is there an intrinsic predictability limit for training trajectories?
- How should an autonomous researcher respond when its forecast is unreliable?
3 comments
Thanks for your review! (2/2)
W2: Curated and community runs
To separate the factors you list, we ran one analysis for each. Logging quality accounts for most of the gap in the error floor, and the loss/RL contrast reappears inside the community set:
- Data quality. Thinning the curated curves to community logging density and adding community-level noise raises the lowest error that a smooth hindsight fit can reach (the error floor) from 0.090 to 0.214, against 0.225 on the community runs (Appendix D.7).
- Training algorithm. Inside the community set, the error relative to repeating the last value is 0.55-0.64 on the 140 fine-tuning and preference-tuning runs and 0.96-1.04 on the 92 RL runs.
- Project familiarity and distribution shift. On 129 runs created after the submission, and after the models' snapshots, by owners outside the benchmark (mostly pretraining losses), the LLMs' error is 0.73-0.74 times that of LC-PFN, the best curve fitter of the paper.
This breakdown is in the new Appendices L.5 and L.6, linked from Appendix D.7.
Q1: Is there an intrinsic predictability limit?
We believe there is, and that it depends on what the forecaster can read. Part of it is randomness that no forecaster can remove: in our sweep, runs that differ only in the random seed already end 0.006 apart in final loss. The rest is information: what happens after the cutoff is often not determined by the curve seen so far.
This second part can move. As discussed in Section 5 of the draft, a better signal may lie in the data, outputs, gradients, and checkpoints that a researcher inspects next, and our agent mode (Section 4.5), where the model opens the run's repository and runs code in a sandbox, is the place to provide them. Current models do not yet turn this access into better forecasts, which motivates agentic post-training as future work: training models to inspect a run as a researcher would. There is room to gain: a curve fitted in hindsight reaches a skill of 0.90 on the 129 new runs, against 0.76 for the best forecaster.
Q2: What should an agent do when its forecast is unreliable?
In our view, a forecast is a signal for a decision, not the decision itself, so the first requirement is that the agent knows how much to trust it. This is why the benchmark asks for quantiles and scores them: the 80% intervals of GPT-5.6 Sol contain the final value 80% of the time, so a wide interval is a usable warning. When the warning fires, the agent should prefer the cheap and reversible action over the irreversible one: keep the run going and look again later, because waiting costs a bounded amount of compute while stopping a good run loses it, or fall back to the simple rule of keeping the runs that are currently ahead. In our 72-run sweep, the leading runs at 20% are closer than the seed noise, and training all runs to 50% before choosing finds the best run, at 62.5% of the full budget instead of 40%. More broadly, we see these forecasts today as signals that assist a human researcher, and an autonomous agent should hand the decision back when its forecast is unreliable.
Both points are now in Section 5.
Thank you for your time and effort in reviewing our paper. We have incorporated your suggested experiments and believe they have strengthened the paper. We hope you could consider increasing your rating in light of these revisions.
Thanks for your review!
Dear Reviewer ip94,
Thank you again for your constructive review. As the discussion period is ending soon, we would greatly appreciate your feedback on our response. Following your comments, we trained two controlled sweeps and replayed the benchmark's sweeps to connect forecasting to decisions (new Section 4.6 and Appendix K), analyzed 129 later runs and the community runs to separate the factors behind the gap (Appendices L.1, L.5, and L.6), and added our view on the predictability limit and on unreliable forecasts to Section 5.
We are happy to answer any further questions. We are grateful for your positive assessment of the paper; if our response has addressed your remaining concerns, we would appreciate your continued support and any update to your score that you find appropriate.
Best regards, Authors
Thanks for your review! (1/2)
Thank you for recognizing that anticipating training outcomes is an important capability missing from current research agents. We ran four new experiments: two 72-run sweeps and a budget replay (W1), and 129 later runs (W2).
W1: From forecasting to experimental decisions
As you point out, Tea Leaves targets one capability: anticipating training outcomes before experiments finish. This decides our setting: each run is forecast from its first 20% and scored on its own, with a skill score (0 for repeating the last observed value, 1 for a perfect forecast). We agree that showing the link to decisions strengthens the paper. We find that forecasting skill and decision quality are positively related but not identical: the models choose the better of two runs before the curves separate, and on pretraining sweeps they need a third less training budget than keeping the current leader (the run that is ahead at 20%); on fine-tuning and RL sweeps, their skill is not yet sufficient to improve on it.
Both studies are now in a new Section 4.6, with full tables in a new Appendix K. Additions are in blue; the submitted numbering is unchanged.
(Our response continues in the next comment with W2, Q1 and Q2.)