acceptodds
Under review as a conference paper at ICLR 2027

Learning Scientific Exploration from Human Research Decisions Trajectories

Abstract

A key challenge in building AI systems for scientific research is enabling : the systematic process of investigating unknown phenomena or ideas to gain new knowledge through sequences of research decisions and actions. Yet this process is largely missing from existing scientific corpora; for example, research papers primarily record final outcomes rather than the trajectories that produced them. In this work, we introduce , a dataset of human research trajectories constructed from Git repositories, where commit histories serve as proxies for intermediate research decisions. We develop an automated and scalable pipeline that extracts structured research trajectories from repository commits, capturing successive changes to methods, experiments, and ablations. We characterize the resulting dataset and show that these trajectories contain meaningful signals about intermediate research decisions beyond what final papers reveal. We further demonstrate utilities of ResearchTrails in multiple use cases, including retrieving human research experience as external skills at test time and training models on research trajectories to improve generalization to new research decisions. Our results suggest a path toward AI systems that learn not only from the products of science, but from the evolving process of discovery itself.

Primary area:
datasets and benchmarks
Keywords:
Datasets, Autoresearch, AI for scientific discovery

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Tne9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes to use the git history of ML research repositories as a proxy for the (scientific) exploration process that precedes a paper. From ~15K NeurIPS 2022-2025 papers they end up with 599 paper/repo pairs that have a "progressive" pre-publication commit history (>=5 commits, >=30 days). An LLM (GPT-5.6 Sol) scans the commits chronologically, groups them into "research decisions" and cites commit IDs (SHAs) and diff excerpts as evidence. The citations are checked syntactically against the stored diffs. This yields 13,271 decisions, about 15 per project on average.

The dataset is then used for a next-decision prediction task: given the first 1-3 decisions of a held-out project, predict the next one. An LLM judge scores the prediction against the real decision on component and operation match (0-4). They compare demonstrations vs. skills distilled from trajectories (vs. from final papers, vs. temporally shuffled trajectories) with Gemini 3.1 Flash-Lite as predictor, and additionally do SFT + GRPO on Qwen3-8B with the judge score as reward. Skills from trajectories beat the other harnesses (1.70 vs 1.26 baseline at prefix 1), and RL lifts the Qwen judge score from 1.20 to 1.51.

Critical strengths
  1. The basic idea is good and, as far as I can tell, new in this form: the git history is the one place where the dead ends of a project are still visible (at least if the authors included the history), and papers systematically remove them. The pipeline design is careful with respect to the multitude of elements: many-to-many commit/decision mapping, open threads for undecided commits, forks handled, notebooks stripped of outputs, every decision must cite an exact diff substring or it is dropped. Table 1 gives an honest funnel, including the fact that only ~4% of papers survive. The two full annotation examples in Figs. 11-14 are convincing as a qualitative sanity check. The Curvature-Tuning trajectory in particular reads like a real project.

  2. The "shuffled skill" and "final paper skill" controls are the correct ones to run. That shuffling the temporal order inside each project drops the score to roughly baseline level is the most interesting single number in the paper, since it suggests the order of decisions carries the signal rather than the content of the decisions. This is the one piece of evidence that actually speaks to the title.

Critical weaknesses
  1. There is no human validation of the extracted decisions at all. The whole dataset is produced by one LLM, and the only "verification" (Sec. 3.3) is that cited SHAs, paths and diff excerpts exist. That checks that the model did not hallucinate evidence, it says nothing about whether the grouping into decisions, the category label, or in particular the outcome label (retained/superseded/abandoned) is correct. For a dataset paper this is the central claim, and currently the "76.3% retained / 11.5% abandoned / 12.1% superseded" statistics and the "research patterns" in Fig. 4 rest entirely on the annotator. A modest study (two people, a few hundred decisions, agreement on category and outcome plus a precision estimate for "is this a decision at all") would go a long way and is not expensive.

  2. The evaluation measures imitation of what the authors did, not whether a suggestion is sensible, and the paper partly concedes this (l. 330). That would be acceptable if the effect sizes were large, but they are not: all scores live between 1.2 and 1.7 on a 0-4 scale, i.e. on average the prediction hits neither the component nor the operation fully, and several rows in Fig. 6 are within one std of each other (3 runs). The Final-paper-skill row goes 1.50 -> 1.23 -> 1.27 across prefix lengths, which does not look like a stable effect. The user simulations (Fig. 8) are two hand-picked examples. The RL section is on 125 trajectories / 375 samples with LoRA rank 8 and the judge as reward, so the 1.20 -> 1.51 gain is on exactly the metric that was optimised. The conclusion's claim that the workflow is "a scalable approach for providing learning supervision" is not supported at this scale.

Questions for authors
  1. Did anyone with domain knowledge check a sample of the 13K decisions? If yes, what were the numbers (agreement on category, on outcome, fraction of "decisions" that are not decisions)? This is the question that decides my score.
  2. 19 vs 125 training trajectories: please clarify the splits for 5.1 and 5.2, and whether the 134 evaluation repos overlap with anything the skills (distilled by Opus 5) or Sol have seen.
  3. Have you run the judge-based evaluation with a judge from a different model family, at least for the main Fig. 6 comparison and for the post-RL Qwen model? Given that Sol is annotator, reward and judge, this seems necessary.
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for the detailed feedback, and we address the concerns and clarification questions as follows.

[Annotation audit] We thank the reviewer for mentioning this. We follow the reviewer’s suggestion and examine the annotated trajectories by three auditors: Opus 5 and two ML researchers with ongoing or defended PhD degree levels. Each auditor audits every decision in 8 blindly sampled trajectories distributed across NeurIPS years {22,23,24,25}, obtaining 383 decisions in total. Each auditor sees the decision text, the cited commit evidence, and the full trajectory, and labels whether the decision is a genuine research decision, its category (method/experiment/ablation), and its outcome (abandoned/superseded/retained). An auditor sees the definition of category and outcome but does not see the annotation pipeline and instruction prompts. A decision can appear superseded/abandoned only after later decisions are examined by the auditor, so an auditor can edit the labels of previously examined decisions. Here is an overview of each auditor's judgements:

Original: 5.6 SolOpus 5 auditorHuman auditor 1Human auditor 2
Total383 decisions372 genuine370 genuine378 genuine
Method94 (64%)97 (26%)98 (26%)87 (23%)
Experiment244 (64%)245 (66%)242 (65%)248 (66%)
Ablation45 (12%)30 (8%)30 (8%)43 (11%)
Retained285 (75%)261 (70%)265 (72%)280 (74%)
Superseded52 (14%)51 (14%)48 (13%)52 (14%)
Abandoned46 (12%)60 (16%)57 (15%)46 (12%)

We observe three takeaways. First, 97% of extracted decisions are genuine research decisions agreed by three auditors, with the remaining identified as engineering-only artifacts. 84% of the genuine decisions get exactly the same category and outcome as the original annotation. Second, 'Ablation' is the least stable category: For example, as audited by Opus, 12 are relabeled as 'experiment', and 5 become ‘method’. Third, human-LLM agreement is 98.8% on category and 98.6% on outcome, against inter-human agreement of 93.8% and 93.8%.

This shows that the research decisions extracted by our pipeline are mostly agreed with by humans and advanced LLMs. Further, it is compatible with adding multiple auditors to improve data quality, since individuals tend to agree on whether a 'decision' is a genuine research decision, as well as its category and outcome. We have added the auditing results and the detailed confusion matrices to Appendix A.2 in our revision.

[Evaluation metric] We agree that matching the human's next decision does not mean a different suggestion is bad. We choose to match the human’s next decision because it is the type of proxy objective commonly used in behavior cloning in robotics [1] and next-token prediction, and the only signal grounded in what human researchers did rather than in a model's opinion.

We now also report additional metrics to measure decision quality generated during evaluation: 'feasible' (whether the suggestion is understandable and actionable) and 'diagnostic' (whether the suggestion improves understanding of the previous confusion and bottleneck). Three evaluators, including a PhD-level human ML researcher, GPT-5.6 Sol, and Opus 5, grade {0,1,2} on each metric, yielding a total score in the range 0-4. The results are as follows and are added in Tables 12-13 in the revision.

Quality score (feasible + diagnostic) of predictions made by different methods.

GPT-5.6 SolOpus 5Human
Baseline2.465 (.071)2.592 (.082)2.513 (.101)
All demos2.188 (.057)2.254 (.079)2.208 (.074)
Random two demos1.983 (.126)2.016 (.119)1.981 (.120)
Retrieve two demos3.211 (.076)3.298 (.049)3.273 (.096)
Shuffled skill2.275 (.078)2.394 (.065)2.016 (.067)
Final paper skill1.969 (.042)2.095 (.033)1.988 (.097)
Skills3.408 (.082)3.512 (.063)3.424 (.082)
GPT-5.6 SolOpus 5Human
Qwen3-8B base1.907 (.051)2.010 (.046)1.883 (.079)
SFT epoch 32.248 (.032)2.460 (.051)2.337 (.024)
GRPO epoch 72.989 (.053)3.026 (.068)3.041 (.072)

We observe that across all evaluation queries where evaluators agree on the ranking of Skills, Skills wins or ties in 96% of cases. We also observe the same performance ordering that "Our Skills > Final-paper skill > Baseline" and "RL checkpoint > SFT checkpoint > Qwen3-8B initialization". Therefore, training on human decision trajectories improves scientific exploration ability rather than simply fitting to the decision-similarity rubrics.

[1] ALVINN: An Autonomous Land Vehicle in a Neural Network. NeurIPS 1988.

AuthorsOpenReview ↗

Rebuttal by Authors (cont.)

[Evaluation scale and significance] We would like to note that a score of 4 requires the exact component and the exact operation of one specific decision based on 1-3 prefix decisions. The low score of the baseline (Table 2) shows that it is a challenging task even for frontier models, yet our methods can improve upon it. Furthermore, we let an ML PhD student predict the next decision for 50 randomly sampled evaluation prefixes with length 3, and the predictions are scored by the same judges. The human predictions get 1.674 by GPT-5.6 Sol and 1.906 by Opus 5. As a comparison, the Gemini predictor with Skills gets 1.607 by Sol and 1.813 by Opus, which is near the human score. Therefore, the relatively low scores reflect task difficulty rather than weak methods.

For statistical significance, we clarify that the ±\pm in Fig. 6 is the standard deviation over 3 runs, each run averaging 134 repositories, rather than the per-prediction spread. We obtain Skills > Baseline by 0.44 / 0.27 / 0.16 at prefixe-length = 1-3, Skills > Final-paper skill by 0.20 / 0.33 / 0.34, Retrieve > Random by 0.12 / 0.24 / 0.22. All nine improvements are statistically significant, with bootstrap 95% confidence intervals excluding zero. All three orderings also hold under the additional judge, Opus 5, as we present below and added in Table 8-11 in revision.

Decision Similarity Score of Gemini predictor judged by Opus 5.

MethodPrefix-1Prefix-2Prefix-3
Baseline1.473 (.088)1.555 (.082)1.619 (.049)
All demos1.537 (.033)1.577 (.049)1.602 (.057)
Random two demos1.460 (.062)1.505 (.038)1.455 (.110)
Retrieve two demos1.590 (.033)1.687 (.068)1.714 (.044)
Shuffled skill1.575 (.022)1.627 (.071)1.555 (.062)
Final paper skill1.764 (.030)1.490 (.089)1.445 (.024)
Skills1.893 (.037)1.781 (.035)1.813 (.042)

Decision Similarity Score of Qwen3-8B at different stages, judged by GPT-5.6 Sol and Opus 5.

ModelGPT-5.6 SolOpus 5
Qwen3-8B base1.0251.219
SFT epoch 31.1991.333
GRPO epoch 71.5051.674
GRPO epoch 101.5101.637

[Final-paper skill] We thank the reviewer for noticing that the judge GPT 5.6-Sol yields the pattern 1.50 →\rightarrow 1.23 →\rightarrow 1.27. We also show that Opus 5 yields a similar pattern (1.76 →\rightarrow 1.49 →\rightarrow 1.45, Table 10). We think it agrees with our intuition that, when the final paper is given, it is relatively easier to predict the first/second step compared to the mid-stage exploration process. Consistent with this, we observe that when the target decision was later superseded or abandoned (which is thus absent from the published papers), the gap between Final-paper skill and Skills tends to be larger (0.424) compared with the gap on retained targets (0.207). This further supports the argument that intermediate decisions carry signals that final papers omit.

[User simulations] The user simulations in Figure 8 are illustrative, and in the rebuttal, we have added additional evaluation of decision quality ('feasible' and 'diagnostic') on the 134 held-out evaluation queries. We asked three evaluators (including a PhD-level human ML researcher, GPT-5.6 Sol, and Opus 5) to grade {0,1,2} on each metric. Detailed setups are described in Tables 12-13 in the revised paper. We observe that across all evaluation queries where evaluators agree on the ranking of Skills, Skills wins or ties in 96% of cases. We also observe the same performance ordering that "Skills > Final-paper skill > Baseline" and "RL checkpoint > SFT checkpoint > Qwen3-8B initialization". Therefore, training on human decision trajectories improves scientific exploration ability rather than simply fitting to the decision-similarity rubrics.

[The "scalable" claim] We thank the reviewer for mentioning this. We would like to clarify that we meant that the supervision pipeline is scalable for three reasons: First, it automatically extracts research decisions from Git commit histories, without requiring human manual labeling of the intermediate research decisions. Second, we show that 4% of NeurIPS-indexed papers contain progressive commit histories, which is a decent percentage and allows for scaling up. Third, our pipeline uses multiple filtering rules and thus does not consume LLM calls until the last stage, so the cost is economical.

Regarding the current dataset curation, we process 15K papers, yielding ~600 papers with progressive commit histories that contain 13K research decisions. Our RL stage is trained on 125 trajectories, which is a proof of concept and shows that the research pattern is learnable--another piece of evidence that scaling up the dataset and model will learn scientific explorations even better. We have made the claims more precise and clear in our revision.

AuthorsOpenReview ↗

Rebuttal by Authors (cont.)

[Train and eval split] We thank the reviewer for pointing this out. We clarify that Section 5.1 uses 19 repositories as the pool for demonstrations and skill distillation. The skills are distilled by Opus 5 from the 19 training trajectories only, and no evaluation trajectory is shown to Opus 5.

Section 5.2 uses 125 repositories ×\times 3 = 375 prefix samples for SFT and RL training, where 19 are a subset of the 125 trajectories. Both Sections 5.1 and 5.2 evaluate on the same 134 held-out repositories, which are disjoint from both training pools. In other words, they do not overlap with any skills the LLMs see, or the trajectories used for training. We have clarified these in Appendix B in the revision.

[More judge model] We thank the reviewer for this question, and we have additionally used Opus 5 as judge, which is a different model family not used as an annotator, reward, or predictor. The results are reported in Tables 8-11, where conclusions from Sol and Opus are consistent: First, both judges have small standard deviations and large Kappa (high agreement) in their judgments. Second, both judges give intended scores on different types of decisions: actual decisions, faithful paraphrases of decisions, adjacent, and unrelated decisions. This shows that our judge rubric works with both judges as desired. Third, under both Sol and Opus judges, we observe that Skills is best at every prefix, Retrieve outperforms Random, Skills outperforms Final-paper skill, and the two-stage SFT+RL training improves Qwen3-8B.

Reviewer_t6vqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a dataset that extracts structured research decisions from repository histories using LLMs. The authors further investigate two applications of these trajectories. At test time, they use human research trajectories as demonstrations or distill them into reusable research skills to improve next-decision prediction. At training time, they fine-tune Qwen3-8B using supervised learning followed by GRPO with decision-similarity rewards. The paper also processes 15K papers from NeurIPS 2022–2025 according to this venue’s relatively broad topic coverage and abundant indices of arXiv links on Semantic Scholar.

Critical strengths
  1. A valuable and meaningful dataset for studying research decision processes. The paper addresses an important limitation of existing scientific research datasets: their lack of decision process. Extracting structured research decisions from GitHub commit histories is an interesting and scalable approach to partially recovering this missing information.
  2. Scale of dataset: The resulting dataset contains 13,271 decisions from 599 research projects and categorizes it, which could support future research on scientific decision-making and research-agent training.
  3. Interesting downstream results. The downstream experiments are meaningful and demonstrate the practical utility of the proposed research trajectories. The authors evaluate both test-time approaches, including demonstrations and distilled research skills, and training-time approaches using SFT and GRPO. The improvements over the corresponding baselines provide encouraging evidence that human research trajectories contain useful information beyond final research papers and can help models make more informed research decisions.
Critical weaknesses
  1. Improvement of decision-similarity optimization. The RL stage optimizes decision-similarity rewards, and the evaluation uses the same type of metric. The answer might come from reward hacking and may not reflect the quality of scientific research.
  2. Potential knowledge leakage. Could the authors clarify the relevant training-data cutoffs and provide a temporally controlled evaluation or contamination analysis?
  3. Decision similarity does not necessarily reflect scientific exploration ability. The paper does not report human evaluations of the scientific usefulness or quality of the generated decisions. Scientific exploration is an open-ended question, there are many dimensions of scientific exploration/usefulness.
Questions for authors

See weakness.

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for the valuable time and feedback, and we address the concerns and clarification questions as follows.

[Evaluation metric as reward] We would like to clarify that using the evaluation criterion as the RL reward is a standard setting of RLHF and RLVR (correctness rewards for math and code; rubric rewards for deep research [1,2], or even in next-token prediction (i.e., token probability is both the training objective and evaluation metric). Apart from the query/response visualizations (Figure 10 in our original submission and revision) that confirm the trained model indeed proposes diagnostic scientific suggestions rather than output hacking tokens, we specify the following evidence to guard against reward hacking:

First, we provide additional judge results. Training rewards come from GPT-5.6 Sol, so we add evaluation with judge Opus 5, which is from a held-out model family (Table 11 in revision). Opus-judged scores rise from 1.22 (base) to 1.33 (SFT) to 1.67 (GRPO), the same trend as the Sol-judged 1.03 →\rightarrow 1.20 →\rightarrow 1.51. A policy exploiting Sol-specific preferences would not transfer to a different judge family.

Decision Similarity Score of Qwen3-8B at different stages judged by GPT-5.6 Sol and Opus 5.

ModelGPT-5.6 SolOpus 5
Qwen3-8B base1.0251.219
SFT epoch 31.1991.333
GRPO epoch 71.5051.674
GRPO epoch 101.5101.637

Second, we evaluate on held-out repositories. All 134 evaluation repositories are disjoint from training, so the reward cannot be memorized.

Furthermore, we add additional metrics for decision quality. We let three evaluators judge how 'feasible' (whether the suggestion is understandable and actionable) and 'diagnostic' (whether the suggestion improves understanding of the previous confusion and bottleneck) a decision is. Specifically, a PhD-level human ML researcher, GPT-5.6 Sol, and Opus 5 grade scores {0,1,2} for each metric, so the total score has range 0-4. The results are as follows and are added in Tables 12-13 in the revision.

Quality scores (feasible + diagnostic) of predictions made by different checkpoints.

GPT-5.6 SolOpus 5Human
Qwen3-8B base1.907 (.051)2.010 (.046)1.883 (.079)
SFT epoch 32.248 (.032)2.460 (.051)2.337 (.024)
GRPO epoch 72.989 (.053)3.026 (.068)3.041 (.072)

Under additional metrics, we observe the same performance ordering: RL checkpoint > SFT checkpoint > Qwen3-8B initialization. Therefore, our two-stage training improves scientific exploration ability rather than simply fitting to rubrics,

[1] DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research. ICML 2026.
[2] Training AI Co-Scientists Using Rubric Rewards. ICML 2026.

[Knowledge leakage] Thank you for this question. We would like to clarify that leakage is prevented by construction, since the predictor receives only the instruction prompt and the preceding decisions; It never sees the title, abstract, authors, or repository URL. The decision texts are abstractions produced by our pipeline and do not exist in the pretraining corpus; recovering the next decision from memory would require recalling the repository's pre-publication commit sequence, not the published paper. The low Baseline scores (1.26-1.45 out of 4 at prefixes 1-3) and the fact that harness yield gains (Skills vs. Shuffled skill vs. Final-paper skill, Retrieve demo vs. Random demo vs. All demo) also argue against recall.

AuthorsOpenReview ↗

Rebuttal by Authors (cont.)

[Additional metric] We agree that similarity to the human's actual next decision is one proxy metric of scientific exploration ability among several. We choose it because it is the only signal grounded in what human researchers do rather than in a model's opinion. It plays the role that demonstrations play in behavior cloning or the token-similarity in next-token prediction.

Furthermore, we would like to note our main contribution as proposing a scalable route for obtaining human demonstration data for improving research capabilities, instead of proposing training methods or presenting best-performing auto-research models. We hope the research community can build upon this dataset, apply and scale up our pipeline, and develop advanced algorithms to train frontier models.

However, we still augment our work with human evaluation, and we agree that a sensible but different suggestion can score low on similarity. We evaluate the predicted next decisions on additional metrics, including ‘feasible’ (whether the suggestion is understandable and actionable) and ‘diagnostic’ (whether the suggestion improves understanding of the previous confusion and bottleneck). We let three evaluators, including a PhD-level human ML researcher, GPT-5.6 Sol, and Opus 5 grade {0,1,2} on each metric, yielding a total score in the range 0-4. The results are as follows and are added in Tables 12-13 in the revision.

Quality score (feasible + diagnostic) of predictions made by different methods.

GPT-5.6 SolOpus 5Human
Baseline2.465 (.071)2.592 (.082)2.513 (.101)
All demos2.188 (.057)2.254 (.079)2.208 (.074)
Random two demos1.983 (.126)2.016 (.119)1.981 (.120)
Retrieve two demos3.211 (.076)3.298 (.049)3.273 (.096)
Shuffled skill2.275 (.078)2.394 (.065)2.016 (.067)
Final paper skill1.969 (.042)2.095 (.033)1.988 (.097)
Skills3.408 (.082)3.512 (.063)3.424 (.082)
GPT-5.6 SolOpus 5Human
Qwen3-8B base1.907 (.051)2.010 (.046)1.883 (.079)
SFT epoch 32.248 (.032)2.460 (.051)2.337 (.024)
GRPO epoch 72.989 (.053)3.026 (.068)3.041 (.072)

We observe that out of all the evaluation queries, conditioned on evaluators' agreement, Skills wins/ties 96% of the time. We also observe the same performance ordering that "Skills > Final-paper skill > Baseline" and "RL checkpoint > SFT checkpoint > Qwen3-8B initialization". Therefore, training on human decision trajectories improves scientific exploration ability rather than simply fitting to the decision-similarity rubrics.

Reviewer_AXzGOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ResearchTrails, a dataset of research decision trajectories extracted from the Git histories of NeurIPS 2022-2025 papers. An LLM pipeline groups commits into method, experiment, and ablation decisions. Models then predict a project's next decision from the earlier ones, and an LLM judge scores the predictions. Skills distilled from trajectories help Gemini 3.1 Flash-Lite more than demonstrations or skills distilled from final papers, and SFT plus GRPO improves Qwen3-8B.

Critical strengths
  • Using commit histories as traces of the research process is a new and well-motivated data source. It keeps abandoned and superseded decisions that papers do not show.
  • The extraction pipeline is carefully engineered, with evidence citations checked against the stored diffs and filtering documented stage by stage (Table 1). The test-time study also includes useful baselines (e.g., shuffled-order and final-paper skills).
Critical weaknesses
  1. The pipeline depends heavily on LLM-based extraction (e.g., the decisions and their labels). There is no evaluation of extraction quality, such as a human audit of a sample of decisions. The judge is checked only against its own reruns and synthetic paraphrases (Tables 6-7).
  2. Following 1, it's not clear whether the evaluation measures research ability rather than fit to the evaluation rubric. The distilled skill (Appendix, pages 23-24) reads like a guide to the label format and judge rubric.
  3. (Minor) The dataset covers only one ML venue.
Questions for authors

See weaknesses.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for the insightful comments, and we address the concerns and clarification questions as follows.

[Annotation audit] We thank the reviewer for mentioning this. We follow the reviewer's suggestion and examine the annotated trajectories by three auditors: Opus 5 and two human ML researchers with ongoing or defended PhD degree levels. Each auditor audits every decision in 8 blindly sampled trajectories distributed across NeurIPS years {22,23,24,25}, 2 in each year, obtaining 383 decisions in total. Each auditor sees the decision text, the cited commit evidence, and the full trajectory, and labels whether the decision is a genuine research decision, its category (method/experiment/ablation), and its outcome (abandoned/superseded/retained). An auditor sees the definition of category and outcome but does not see the annotation pipeline or instruction prompts. A decision can appear superseded or abandoned only after later decisions are examined by the auditor, so auditors can edit the judgements of previously examined decisions. Here is an overview of each auditor's judgements:

Original: 5.6 SolOpus 5 auditorHuman auditor 1Human auditor 2
Total383 decisions372 genuine370 genuine378 genuine
Method94 (64%)97 (26%)98 (26%)87 (23%)
Experiment244 (64%)245 (66%)242 (65%)248 (66%)
Ablation45 (12%)30 (8%)30 (8%)43 (11%)
Retained285 (75%)261 (70%)265 (72%)280 (74%)
Superseded52 (14%)51 (14%)48 (13%)52 (14%)
Abandoned46 (12%)60 (16%)57 (15%)46 (12%)

We observe three takeaways. First, 97% of extracted decisions are genuine research decisions agreed by three auditors, with the remaining identified as engineering-only artifacts. 84% of the genuine decisions get exactly the same category and outcome as the original annotation. Second, 'ablation' is the least stable category: For example, as audited by Opus, 12 are relabeled as 'experiment', and 5 become ‘method’. Third, human-LLM agreement is 98.8% on category and 98.6% on outcome, against inter-human agreement of 93.8% and 93.8%.

This shows that the research decisions extracted by our pipeline are mostly agreed with by humans and advanced LLMs. Further, it is compatible with adding multiple auditors to improve data quality, since individuals tend to agree on whether a 'decision' is a genuine research decision, as well as its category and outcome. We have added the auditing results and the detailed confusion matrices to Appendix A.2 in our revision.

[More judge model] We thank the reviewer for the suggestion. We additionally use Opus 5 as the judge, which is a different model family not used as an annotator, reward, or predictor. We have also added two evaluations of judge quality. The results are reported in Tables 8-11, where conclusions from Sol and Opus are consistent: First, both judges have small standard deviations and large Kappa (high agreement) in their judgments. Second, both judges give intended scores on different types of decisions: actual decisions, faithful paraphrases of decisions, adjacent, and unrelated decisions. This shows that our judge rubric works with both judges as desired. Third, under both Sol and Opus judges, we observe that Skills is best at every prefix, Retrieve outperforms Random, Skills outperforms Final-paper skill, and the two-stage SFT+RL training improves Qwen3-8B.

Standard deviation and Kappa of the judge scores measured on 100 baseline predictions.

Judge modelMetricSample SDFleiss' Kappa
GPT-5.6 SolScore0.0320.719
Component0.0210.816
Operation0.0120.800
OpusScore0.0060.691
Component0.0350.831
Operation0.0400.778

Judges' scores match the intended scores when evaluating the actual decisions, their faithful paraphrases, adjacent decisions, and unrelated decisions.

Judge modelPredicted decisionIntended scoreScoreComponentOperation
GPT-5.6 SolSame44.002.002.00
Paraphrase44.002.002.00
Adjacent22.391.341.05
Unrelated00.100.040.06
Opus 5Same44.002.002.00
Paraphrase44.002.002.00
Adjacent22.681.421.26
Unrelated00.120.040.08

Decision similarity of the predicted next decision on previous 1/2/3 decisions. Scores are judged by Opus 5.

MethodPrefix-1Prefix-2Prefix-3
Baseline1.473 (.088)1.555 (.082)1.619 (.049)
All demos1.537 (.033)1.577 (.049)1.602 (.057)
Random two demos1.460 (.062)1.505 (.038)1.455 (.110)
Retrieve two demos1.590 (.033)1.687 (.068)1.714 (.044)
Shuffled skill1.575 (.022)1.627 (.071)1.555 (.062)
Final paper skill1.764 (.030)1.490 (.089)1.445 (.024)
Skills1.893 (.037)1.781 (.035)1.813 (.042)

Decision Similarity Score of different checkpoints.

ModelGPT-5.6 SolOpus 5
Qwen3-8B base1.0251.219
SFT epoch 31.1991.333
GRPO epoch 71.5051.674
GRPO epoch 101.5101.637
AuthorsOpenReview ↗

Rebuttal by Authors (cont.)

[Fit to evaluation metric] We thank the reviewer and agree that it helps to distinguish improved research exploration ability from fit to the evaluation rubric. We ensure that our skill construction process never sees the output schema or the judge rubric (Page 21-22 in revision).

We also note that our Skills differ from Shuffled skills and Final-paper skills in terms of both the content (Appendix C in the revision) and the evaluation scores. For example, Skills’ Step-2 gap rules (e.g., "data present, measurement absent →\rightarrow define how outcomes are scored"; "mechanism just introduced with one salient knob →\rightarrow sweep it") are claims about examining how research projects proceed, not about the rubric. Shuffled skill and Final-paper skill score 0.33 and 0.20-0.37 below Skills under Sol judge (Table 2), and 0.26-0.32 and 0.13-0.37 below under Opus 5 (Table 10). If the rubric fit explained the gains, we would expect those methods to be much more similar.

To further show that Skills improve scientific exploration ability rather than fitting to rubrics, we let three evaluators judge the quality of the predicted decisions in terms of 'feasible' (whether the suggestion is understandable and actionable) and 'diagnostic' (whether the suggestion improves understanding of the previous confusion and bottleneck). Specifically, a PhD-level human ML researcher, GPT-5.6 Sol, and Opus 5 grade each decision with scores {0,1,2} for each metric, so the total score has range 0-4. The results are as follows and are added in Tables 12-13 in the revision.

Quality score (feasible + diagnostic scores) of predictions made by different methods.

GPT-5.6 SolOpus 5Human
Baseline2.465 (.071)2.592 (.082)2.513 (.101)
All demos2.188 (.057)2.254 (.079)2.208 (.074)
Random two demos1.983 (.126)2.016 (.119)1.981 (.120)
Retrieve two demos3.211 (.076)3.298 (.049)3.273 (.096)
Shuffled skill2.275 (.078)2.394 (.065)2.016 (.067)
Final paper skill1.969 (.042)2.095 (.033)1.988 (.097)
Skills3.408 (.082)3.512 (.063)3.424 (.082)
GPT-5.6 SolOpus 5Human
Qwen3-8B base1.907 (.051)2.010 (.046)1.883 (.079)
SFT epoch 32.248 (.032)2.460 (.051)2.337 (.024)
GRPO epoch 72.989 (.053)3.026 (.068)3.041 (.072)

We observe that across all evaluation queries where evaluators agree on the ranking of Skills, Skills wins or ties in 96% of cases. We also observe the same performance ordering that "Skills > Final-paper skill > Baseline" and "RL checkpoint > SFT checkpoint > Qwen3-8B initialization". Therefore, training on human decision trajectories improves LLMs' scientific exploration ability rather than simply fitting to the decision-similarity rubrics.

[Beyond ML venue] The proposed pipeline needs only paper/repository pairs with pre-publication commit histories, so it applies to any computation-driven field whose exploration is recorded in code. We illustrate it by applying our pipeline to the MLSys conference and obtain 247 indexed papers →\rightarrow 17 annotated trajectories with 598 decisions (49.3% method, 37.6% experiment, 13.0% ablation; 79.1% retained, 7.7% abandoned, 13.2% superseded). We have uploaded the trajectories to an anonymous repository (see link in the global official comment).

Thank you for the responses. I think the annotation audit experiment is a good augmentation to the paper. I remain supportive of the paper and my score remains at 3

Comments on the paper

1 comment

Dear reviewers, as the discussion period is coming to a close, could you please review the authors’ rebuttal and briefly comment on whether it addresses your concerns? If appropriate, please also update your scores accordingly. Thank you for your time and contributions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.