- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes to use the git history of ML research repositories as a proxy for the (scientific) exploration process that precedes a paper. From ~15K NeurIPS 2022-2025 papers they end up with 599 paper/repo pairs that have a "progressive" pre-publication commit history (>=5 commits, >=30 days). An LLM (GPT-5.6 Sol) scans the commits chronologically, groups them into "research decisions" and cites commit IDs (SHAs) and diff excerpts as evidence. The citations are checked syntactically against the stored diffs. This yields 13,271 decisions, about 15 per project on average.
The dataset is then used for a next-decision prediction task: given the first 1-3 decisions of a held-out project, predict the next one. An LLM judge scores the prediction against the real decision on component and operation match (0-4). They compare demonstrations vs. skills distilled from trajectories (vs. from final papers, vs. temporally shuffled trajectories) with Gemini 3.1 Flash-Lite as predictor, and additionally do SFT + GRPO on Qwen3-8B with the judge score as reward. Skills from trajectories beat the other harnesses (1.70 vs 1.26 baseline at prefix 1), and RL lifts the Qwen judge score from 1.20 to 1.51.
Critical strengths
-
The basic idea is good and, as far as I can tell, new in this form: the git history is the one place where the dead ends of a project are still visible (at least if the authors included the history), and papers systematically remove them. The pipeline design is careful with respect to the multitude of elements: many-to-many commit/decision mapping, open threads for undecided commits, forks handled, notebooks stripped of outputs, every decision must cite an exact diff substring or it is dropped. Table 1 gives an honest funnel, including the fact that only ~4% of papers survive. The two full annotation examples in Figs. 11-14 are convincing as a qualitative sanity check. The Curvature-Tuning trajectory in particular reads like a real project.
-
The "shuffled skill" and "final paper skill" controls are the correct ones to run. That shuffling the temporal order inside each project drops the score to roughly baseline level is the most interesting single number in the paper, since it suggests the order of decisions carries the signal rather than the content of the decisions. This is the one piece of evidence that actually speaks to the title.
Critical weaknesses
-
There is no human validation of the extracted decisions at all. The whole dataset is produced by one LLM, and the only "verification" (Sec. 3.3) is that cited SHAs, paths and diff excerpts exist. That checks that the model did not hallucinate evidence, it says nothing about whether the grouping into decisions, the category label, or in particular the outcome label (retained/superseded/abandoned) is correct. For a dataset paper this is the central claim, and currently the "76.3% retained / 11.5% abandoned / 12.1% superseded" statistics and the "research patterns" in Fig. 4 rest entirely on the annotator. A modest study (two people, a few hundred decisions, agreement on category and outcome plus a precision estimate for "is this a decision at all") would go a long way and is not expensive.
-
The evaluation measures imitation of what the authors did, not whether a suggestion is sensible, and the paper partly concedes this (l. 330). That would be acceptable if the effect sizes were large, but they are not: all scores live between 1.2 and 1.7 on a 0-4 scale, i.e. on average the prediction hits neither the component nor the operation fully, and several rows in Fig. 6 are within one std of each other (3 runs). The Final-paper-skill row goes 1.50 -> 1.23 -> 1.27 across prefix lengths, which does not look like a stable effect. The user simulations (Fig. 8) are two hand-picked examples. The RL section is on 125 trajectories / 375 samples with LoRA rank 8 and the judge as reward, so the 1.20 -> 1.51 gain is on exactly the metric that was optimised. The conclusion's claim that the workflow is "a scalable approach for providing learning supervision" is not supported at this scale.
Questions for authors
- Did anyone with domain knowledge check a sample of the 13K decisions? If yes, what were the numbers (agreement on category, on outcome, fraction of "decisions" that are not decisions)? This is the question that decides my score.
- 19 vs 125 training trajectories: please clarify the splits for 5.1 and 5.2, and whether the 134 evaluation repos overlap with anything the skills (distilled by Opus 5) or Sol have seen.
- Have you run the judge-based evaluation with a judge from a different model family, at least for the main Fig. 6 comparison and for the post-RL Qwen model? Given that Sol is annotator, reward and judge, this seems necessary.
3 comments
Rebuttal by Authors (cont.)
[Evaluation scale and significance] We would like to note that a score of 4 requires the exact component and the exact operation of one specific decision based on 1-3 prefix decisions. The low score of the baseline (Table 2) shows that it is a challenging task even for frontier models, yet our methods can improve upon it. Furthermore, we let an ML PhD student predict the next decision for 50 randomly sampled evaluation prefixes with length 3, and the predictions are scored by the same judges. The human predictions get 1.674 by GPT-5.6 Sol and 1.906 by Opus 5. As a comparison, the Gemini predictor with Skills gets 1.607 by Sol and 1.813 by Opus, which is near the human score. Therefore, the relatively low scores reflect task difficulty rather than weak methods.
For statistical significance, we clarify that the in Fig. 6 is the standard deviation over 3 runs, each run averaging 134 repositories, rather than the per-prediction spread. We obtain Skills > Baseline by 0.44 / 0.27 / 0.16 at prefixe-length = 1-3, Skills > Final-paper skill by 0.20 / 0.33 / 0.34, Retrieve > Random by 0.12 / 0.24 / 0.22. All nine improvements are statistically significant, with bootstrap 95% confidence intervals excluding zero. All three orderings also hold under the additional judge, Opus 5, as we present below and added in Table 8-11 in revision.
Decision Similarity Score of Gemini predictor judged by Opus 5.
| Method | Prefix-1 | Prefix-2 | Prefix-3 |
|---|---|---|---|
| Baseline | 1.473 (.088) | 1.555 (.082) | 1.619 (.049) |
| All demos | 1.537 (.033) | 1.577 (.049) | 1.602 (.057) |
| Random two demos | 1.460 (.062) | 1.505 (.038) | 1.455 (.110) |
| Retrieve two demos | 1.590 (.033) | 1.687 (.068) | 1.714 (.044) |
| Shuffled skill | 1.575 (.022) | 1.627 (.071) | 1.555 (.062) |
| Final paper skill | 1.764 (.030) | 1.490 (.089) | 1.445 (.024) |
| Skills | 1.893 (.037) | 1.781 (.035) | 1.813 (.042) |
Decision Similarity Score of Qwen3-8B at different stages, judged by GPT-5.6 Sol and Opus 5.
| Model | GPT-5.6 Sol | Opus 5 |
|---|---|---|
| Qwen3-8B base | 1.025 | 1.219 |
| SFT epoch 3 | 1.199 | 1.333 |
| GRPO epoch 7 | 1.505 | 1.674 |
| GRPO epoch 10 | 1.510 | 1.637 |
[Final-paper skill] We thank the reviewer for noticing that the judge GPT 5.6-Sol yields the pattern 1.50 1.23 1.27. We also show that Opus 5 yields a similar pattern (1.76 1.49 1.45, Table 10). We think it agrees with our intuition that, when the final paper is given, it is relatively easier to predict the first/second step compared to the mid-stage exploration process. Consistent with this, we observe that when the target decision was later superseded or abandoned (which is thus absent from the published papers), the gap between Final-paper skill and Skills tends to be larger (0.424) compared with the gap on retained targets (0.207). This further supports the argument that intermediate decisions carry signals that final papers omit.
[User simulations] The user simulations in Figure 8 are illustrative, and in the rebuttal, we have added additional evaluation of decision quality ('feasible' and 'diagnostic') on the 134 held-out evaluation queries. We asked three evaluators (including a PhD-level human ML researcher, GPT-5.6 Sol, and Opus 5) to grade {0,1,2} on each metric. Detailed setups are described in Tables 12-13 in the revised paper. We observe that across all evaluation queries where evaluators agree on the ranking of Skills, Skills wins or ties in 96% of cases. We also observe the same performance ordering that "Skills > Final-paper skill > Baseline" and "RL checkpoint > SFT checkpoint > Qwen3-8B initialization". Therefore, training on human decision trajectories improves scientific exploration ability rather than simply fitting to the decision-similarity rubrics.
[The "scalable" claim] We thank the reviewer for mentioning this. We would like to clarify that we meant that the supervision pipeline is scalable for three reasons: First, it automatically extracts research decisions from Git commit histories, without requiring human manual labeling of the intermediate research decisions. Second, we show that 4% of NeurIPS-indexed papers contain progressive commit histories, which is a decent percentage and allows for scaling up. Third, our pipeline uses multiple filtering rules and thus does not consume LLM calls until the last stage, so the cost is economical.
Regarding the current dataset curation, we process 15K papers, yielding ~600 papers with progressive commit histories that contain 13K research decisions. Our RL stage is trained on 125 trajectories, which is a proof of concept and shows that the research pattern is learnable--another piece of evidence that scaling up the dataset and model will learn scientific explorations even better. We have made the claims more precise and clear in our revision.
Rebuttal by Authors (cont.)
[Train and eval split] We thank the reviewer for pointing this out. We clarify that Section 5.1 uses 19 repositories as the pool for demonstrations and skill distillation. The skills are distilled by Opus 5 from the 19 training trajectories only, and no evaluation trajectory is shown to Opus 5.
Section 5.2 uses 125 repositories 3 = 375 prefix samples for SFT and RL training, where 19 are a subset of the 125 trajectories. Both Sections 5.1 and 5.2 evaluate on the same 134 held-out repositories, which are disjoint from both training pools. In other words, they do not overlap with any skills the LLMs see, or the trajectories used for training. We have clarified these in Appendix B in the revision.
[More judge model] We thank the reviewer for this question, and we have additionally used Opus 5 as judge, which is a different model family not used as an annotator, reward, or predictor. The results are reported in Tables 8-11, where conclusions from Sol and Opus are consistent: First, both judges have small standard deviations and large Kappa (high agreement) in their judgments. Second, both judges give intended scores on different types of decisions: actual decisions, faithful paraphrases of decisions, adjacent, and unrelated decisions. This shows that our judge rubric works with both judges as desired. Third, under both Sol and Opus judges, we observe that Skills is best at every prefix, Retrieve outperforms Random, Skills outperforms Final-paper skill, and the two-stage SFT+RL training improves Qwen3-8B.
Rebuttal by Authors
We thank the reviewer for the detailed feedback, and we address the concerns and clarification questions as follows.
[Annotation audit] We thank the reviewer for mentioning this. We follow the reviewer’s suggestion and examine the annotated trajectories by three auditors: Opus 5 and two ML researchers with ongoing or defended PhD degree levels. Each auditor audits every decision in 8 blindly sampled trajectories distributed across NeurIPS years {22,23,24,25}, obtaining 383 decisions in total. Each auditor sees the decision text, the cited commit evidence, and the full trajectory, and labels whether the decision is a genuine research decision, its category (method/experiment/ablation), and its outcome (abandoned/superseded/retained). An auditor sees the definition of category and outcome but does not see the annotation pipeline and instruction prompts. A decision can appear superseded/abandoned only after later decisions are examined by the auditor, so an auditor can edit the labels of previously examined decisions. Here is an overview of each auditor's judgements:
We observe three takeaways. First, 97% of extracted decisions are genuine research decisions agreed by three auditors, with the remaining identified as engineering-only artifacts. 84% of the genuine decisions get exactly the same category and outcome as the original annotation. Second, 'Ablation' is the least stable category: For example, as audited by Opus, 12 are relabeled as 'experiment', and 5 become ‘method’. Third, human-LLM agreement is 98.8% on category and 98.6% on outcome, against inter-human agreement of 93.8% and 93.8%.
This shows that the research decisions extracted by our pipeline are mostly agreed with by humans and advanced LLMs. Further, it is compatible with adding multiple auditors to improve data quality, since individuals tend to agree on whether a 'decision' is a genuine research decision, as well as its category and outcome. We have added the auditing results and the detailed confusion matrices to Appendix A.2 in our revision.
[Evaluation metric] We agree that matching the human's next decision does not mean a different suggestion is bad. We choose to match the human’s next decision because it is the type of proxy objective commonly used in behavior cloning in robotics [1] and next-token prediction, and the only signal grounded in what human researchers did rather than in a model's opinion.
We now also report additional metrics to measure decision quality generated during evaluation: 'feasible' (whether the suggestion is understandable and actionable) and 'diagnostic' (whether the suggestion improves understanding of the previous confusion and bottleneck). Three evaluators, including a PhD-level human ML researcher, GPT-5.6 Sol, and Opus 5, grade {0,1,2} on each metric, yielding a total score in the range 0-4. The results are as follows and are added in Tables 12-13 in the revision.
Quality score (feasible + diagnostic) of predictions made by different methods.
We observe that across all evaluation queries where evaluators agree on the ranking of Skills, Skills wins or ties in 96% of cases. We also observe the same performance ordering that "Our Skills > Final-paper skill > Baseline" and "RL checkpoint > SFT checkpoint > Qwen3-8B initialization". Therefore, training on human decision trajectories improves scientific exploration ability rather than simply fitting to the decision-similarity rubrics.
[1] ALVINN: An Autonomous Land Vehicle in a Neural Network. NeurIPS 1988.