- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes using off-the-shelf LLMs as Research World Models which can be used to predict the outcome of a proposed change before evaluating it. They find that providing records of past experiments, from the same or even different environments, substantially improve both LLM forecasts accuracy and candidate selection, and helps much more than using a stronger model or increasing reasoning effort.
Critical strengths
- The result that experimental records enhance performance much more than model choice or reasoning effort seems to hold across all 13 models.
- The experimental data is extensive with 2,653 experimental records from nine research environments spanning pretraining, post-training, and inference.
Critical weaknesses
The source of the gains is unclear. The current baselines (TF-IDF kNN and ridge) are weak and cannot establish whether the gains come from LLM reasoning or merely access to the experimental records. Stronger baselines such as embedding-based retrieval or a small model fine-tuned on the same records would help. Ablations on the number of records could also clarify how performance scales with available data.
Questions for authors
- How were the history and test splits constructed for the in-environment setting? Could closely related interventions appear in both, and if so, how can you distinguish whether the gains reflect generalization to new interventions or partly lookup of closely related ones in the context?
- How does the in-environment performance change as the number of experimental records varies, and did you test whether giving the LLM only the few most similar records recovers most of the gain?
4 comments
Dear Reviewer, thank you again for your review. We have posted responses with new analyses addressing your concerns Stronger baselines, ablation of experimental record quantity, and In-env history/test splits . We would be grateful to know whether these address your concerns, and we are happy to clarify anything further before the discussion period ends.
Thank you for the answer and new experiments. The embedding-retrieval comparison addresses my main concern on whether the LLM adds substantial value beyond access to the records. The history-size ablation and the description of the split construction are also helpful. Given the new evidence, I am raising my score.
Thank you for your positive assessment and recognition of our work! We are glad that your concern has been resolved. We also sincerely appreciate the time and effort you devoted to reviewing our work and engaging in the discussion.
W1: Stronger baselines: embedding-based retrieval
We thank the reviewer for this suggestion. We added embedding-based retrieval baseline and evaluated two ways of using the retrieved records: directly averaging their gains (Embedding-top16 gain average, non-LLM baseline), or providing them to the same LLM for prediction (embedding-top16+LLM).
“Embedding-top16 gain average” achieves 0.62 Spearman. Together with the results of kNN (0.78) and Ridge(0.67) baseline, it indicates that merely having “access to the experimental records” is insufficient to reproduce the LLM’s predictive performance (full320+LLM:0.9). In contrast, “embedding-top16+LLM” achieves 0.87, indicating that the gains come from LLM reasoning.
W1 & Q2: ablation of experimental record quantity.
We vary the number of historical records in the OLMo3-100M in-env setting, keeping the test set fixed and using Claude Opus 5 (HIGH).
In this setting, 128 records already capture most of the predictive improvement (0.88 vs 0.9). Further increases in history size yield smaller, non-monotonic improvements, although MAE continues to decrease overall.
Q1: In-env history/test splits and potential gains from looking up related interventions
We explicitly controlled overlap between highly related interventions when constructing the history and test splits. Within each intervention direction, we connected candidates satisfying both a structured TF-IDF similarity ≥0.90 and a typed intervention distance ≤0.10, and assigned each connected component entirely to the same split. History and test sets contain no cross-split pairs of highly related interventions as defined by these criteria. Regarding whether simple lookup can explain the predictive gains, please see W1 ‘stronger baselines’ table. Comparing the rows of ‘Embedding-top1 gain’, ‘Embedding-top16 gain average’ and ‘embedding-top16+LLM’ shows that most gain comes from generalization rather than direct looking up the table.
Q2: 'LLM+most similar records' vs 'LLM+full records'
See ‘W1: Stronger baselines: embedding-based retrieval’ table. Comparing ‘embedding-top16+LLM’ and ‘full320+LLM’ shows that giving the LLM only the few most similar records recovers most of the gain.