acceptodds
Under review as a conference paper at ICLR 2027

Innovation Prior: Learning to Discover by Distilling the Scientist's Mind

Abstract

Large language models (LLMs) excel at tasks that ask for known methods to be applied well, but on research tasks they recombine and retune existing methods instead of inventing new ones as human scientists do. A harness around a frozen model can only select among what the model produces, and reinforcement learning can only strengthen what it already does, so the missing ability has to come from training data. Papers are in that data, but a paper presents a finished result and never shows how its authors thought. We design a pipeline that reconstructs that thinking and use it to build *Innovation Prior*, a dataset of how scientists reasoned their way to their methods. Each reconstruction draws on multiple sources beyond the paper and includes the authors' exploration of routes that never reached the final method. We fine-tune Qwen3.5-4B and 9B on *Innovation Prior*, then run one RL recipe from the fine-tuned model and from the base model. RL from the fine-tuned model outperforms RL from the base on research benchmarks at both sizes, while none of our training data targets these benchmarks. *Innovation Prior* thus sets a prior for discovery that RL then amplifies. Ablations show that both the multiple sources and this exploration are necessary. The advantage carries over to test-time scaling, where the model scores higher for the same compute, and generalizes to research judgment. The model also explores more broadly, both within and across attempts, and case studies show that it reaches simple but effective methods from first principles.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI for Research, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TLkYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Innovation Prior, which reconstructs scientific discovery trace from multi-source materials to capture problem analysis and abandoned design choices, instead of only learning final methods from finished paper. Based on a dataset of 1,185 discovery units and 164 research evolution chains, the authors SFT Qwen3.5-4B and 9B then apply GRPO with 1,300 extra program optimization tasks. Experiments on four benchmarks show Innovation Prior improves RL model performance at both scales, with larger gains for the 4B model.

Critical strengths

1.The paper proposes a concrete, verifiable method for constructing data, using author accounts, the original paper, and review responses as supplemental data to show design backgrounds, alternative plans, and thought processes that the final paper can't display. 2.The paper designs a comprehensive experiments, and directly tests the combined effects of SFT and RL. The results found SFT alone doesn't consistently improve performance, but when combined with RL, the overall results are better than just RL, showing that learning the reconstruction of discovery trace can help subsequent RL.

Critical weaknesses

1.Whether the gains stem from the specific innovation prior, other fine-tuning effects, or RL stability differences remains questionable. The current ablation keeps the C++ data, which helps compare parts of the Innovation Prior, but it still can't completely isolate the source of gains during the whole fine-tuning phase. 2.The claim that reconstructed discovery traces reflect scientists’ reasoning processes requires stronger evidence. It remains to be validated that the resulting gains indicate improved scientific reasoning.

Questions for authors

1.Result stability: The work lacks multiple independent training replicates, task-wise variance and confidence intervals. It remains unclear whether the 4B direct RL degradation is persistent and whether the marginal gains at 9B exceed random fluctuations. 2.Gain attribution: The performance brought solely by C++ data and generic reasoning data with matched token budget remains unreported. It is unclear whether the ablations match the total number of training tokens and the distribution of example lengths. 3.Discovery traces authenticity: Independent human sampling audit is absent. The proportion of genuine author-tried paths versus post-hoc rationalization and reconstructed derivations within key steps is unknown.

5 comments
AuthorsOpenReview ↗

Response to Reviewer TLkY (1/3)

Thank you for your thoughtful review and for recognizing our concrete, verifiable construction method and the comprehensive experiments that test SFT and RL together. We address your concerns below.


Gain Attribution

"...Whether the gains stem from the specific innovation prior, other fine-tuning effects, or RL stability differences remains questionable..."

Thank you for raising this important point. We ran a new experiment, Maintain-only. It fine-tunes the model on only the 338 C++ problems that Innovation Prior includes to maintain coding ability, without any discovery traces, and then runs the same RL; the SFT recipe, merge weight, RL recipe and evaluation are all unchanged. Maintain-only thus shares the fine-tuning step and the C++ data with InnoPrior and differs from it only in the discovery traces.

ModelFrontierCS Alg. mean@5FrontierCS Alg. best@5FrontierCS Res. mean@5FrontierCS Res. best@5ALE-Bench mean@5ALE-Bench best@5MLS-Bench
4B Base4.8912.406.5216.732004132.98
4B Maintain-only4.109.509.9919.562063544.83
4B InnoPrior3.979.606.9915.971963886.63
4B Base + RL0.612.949.0718.269111019.28
4B Maintain-only + RL5.5110.5419.7526.8025647519.74
4B InnoPrior + RL6.1311.7720.8828.6132064419.95
9B Base4.4712.0710.3727.6624353411.77
9B Maintain-only4.5212.9510.5230.362044344.98
9B InnoPrior4.5712.279.9027.762325247.17
9B Base + RL7.5613.2616.3330.2331663416.59
9B Maintain-only + RL7.1315.1416.0429.7026855713.80
9B InnoPrior + RL8.2514.8218.4131.3231766019.52

Before RL, Maintain-only is not consistently stronger than Base, so the C++ problems add little by themselves. After RL, InnoPrior + RL is above Maintain-only + RL on every metric at 4B and on all but one at 9B. Since Maintain-only + RL also trains stably, the gain comes from the discovery traces rather than from other fine-tuning effects or RL stability. We have added Maintain-only to Table 1 and Sections 3.1 and 3.2 of the revised paper.

"...The current ablation keeps the C++ data...it still can't completely isolate the source of gains during the whole fine-tuning phase..."

Thank you for this observation. We agree, and Maintain-only closes exactly this gap: it keeps the C++ data and the whole fine-tuning phase and removes all the traces, with the results shown above.

"...The performance brought solely by C++ data and generic reasoning data with matched token budget remains unreported..."

Thank you for this question. The performance of the C++ data alone is Maintain-only above. For generic reasoning data, our ablations are the matched comparison: they reason about the same methods in the same form, with the sources beyond the paper or the exploration removed, and the table below shows that removing either lowers every metric.

4B, after RLFrontierCS Alg. mean@5FrontierCS Alg. best@5FrontierCS Res. mean@5FrontierCS Res. best@5ALE-Bench mean@5ALE-Bench best@5MLS-Bench
w/o multiple sources + RL5.2210.4720.4728.1924249413.71
w/o fallback patterns + RL6.0911.1719.6826.1825049714.31
InnoPrior + RL6.1311.7720.8828.6132064419.95

"...whether the ablations match the total number of training tokens and the distribution of example lengths..."

Thank you for the careful question about matching. Each ablation rebuilds the same units, so it has the same number of training examples as the full data, and both ablations are trained for the same number of optimizer steps. We have stated this explicitly in Section 3.3 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer TLkY (2/3)

Result Stability

"...lacks multiple independent training replicates, task-wise variance and confidence intervals..."

"...whether the 4B direct RL degradation is persistent..."

Thank you for raising this; we agree that the persistence of the 4B degradation needed to be established, and we have now done so. We focused the reruns on the 4B Base + RL baseline. It trains normally up to step 15, as the table below shows, where scored counts rollouts with a nonzero score, Inside counts rollouts that end inside the reasoning before any answer, At limit counts rollouts that reach the length limit, Length is in tokens, and the last column gives the entropy of RL from the prior.

StepRewardScoredInsideLengthAt limitEntropyKLGrad normEntropy, prior
16.123%0%15.5K5%0.740.240.860.75
28.831%0%16.0K4%0.730.250.660.74
37.523%0%19.7K12%0.740.220.370.76
48.930%0%19.2K18%0.650.210.430.63
58.227%1%18.9K21%0.620.190.330.61
66.325%5%22.5K48%0.430.130.460.56
73.916%4%25.4K61%0.360.130.390.50
85.716%7%25.7K70%0.240.080.310.41
96.022%4%22.4K49%0.340.120.250.50
109.326%3%20.4K44%0.350.140.240.46
116.426%2%20.0K40%0.350.130.210.43
125.924%1%19.3K37%0.350.140.240.50
138.532%1%17.0K29%0.350.160.210.37
149.735%1%16.2K28%0.320.170.260.48
158.432%2%16.5K30%0.300.170.270.46

From the step-15 checkpoint we ran steps 16 to 20 three times, and all three runs collapse in the same way, as the table below shows with the median rollout length in characters. The degradation is therefore persistent: under this recipe, RL from the 4B base cannot be trained stably to step 20. We have added this to Section 3.2 of the revised paper.

RunStepInside reasoningLengthZero rewardMean reward
116 / 17 / 18 / 19 / 202% / 5% / 30% / 58% / 86%35.7K / 33.5K / 26.7K / 18.6K / 9.8K66% / 62% / 60% / 67% / 81%6.9 / 8.9 / 11.2 / 8.3 / 3.4
216 / 17 / 18 / 19 / 202% / 5% / 31% / 68% / 93%35.8K / 34.1K / 29.2K / 18.7K / 9.0K67% / 60% / 56% / 73% / 90%7.5 / 10.8 / 13.1 / 7.7 / 2.4
316 / 17 / 18 / 19 / 202% / 7% / 51% / 62% / 87%38.1K / 33.0K / 27.1K / 17.5K / 9.8K65% / 62% / 71% / 70% / 83%8.2 / 11.9 / 7.5 / 7.5 / 3.8

"...whether the marginal gains at 9B exceed random fluctuations..."

We would like to note that the 9B comparison is not a single noisy number. Each FrontierCS and ALE-Bench score already pools five samples per problem, and the 9B rows of the table in our previous comment show RL from the prior above RL from the base on every metric of all four benchmarks, which is not the pattern of a random fluctuation. Figure 6 further shows that it keeps the lead over many rounds of test-time feedback. Section 3.2 of the revised paper now states this consistency directly.

AuthorsOpenReview ↗

Response to Reviewer TLkY (3/3)

Faithfulness of the Traces and Scientific Reasoning

"...The claim that reconstructed discovery traces reflect scientists' reasoning processes requires stronger evidence..."

"...Independent human sampling audit is absent. The proportion of genuine author-tried paths versus post-hoc rationalization...is unknown..."

Thank you for raising this important point. The faithfulness of the traces is central to our data, and we agree that it deserves a direct answer. We control it during construction and audit it afterward, unit by unit.

During construction, a fallback enters a trace only if the records show that the authors considered or tried it (Lines 209–210), a trace may not report the outcome of an experiment the narrator claims to run, and an adversarial verifier checks the key step of every trace against the sources (Appendix D). The agent also records in notes which sources it collected and which steps each of them supports.

After construction, a separate audit checks every trace against these notes and the source files rather than trusting the construction agent: a source counts as used only if the auditor can point to the step that depends on it, and labels of low confidence get a second reading. The table below gives, for each kind of source, the number of discoveries whose trace uses it under this audit, and we have added this procedure to Section 2.1 of the revised paper.

Kind of sourceAncestor worksCodeThird-party explanationsSelf-accounts
Discoveries whose trace uses it, of 1,185871546340221

The data then went through several more rounds in which agents re-audited the units under the same rules and improved those that fell short, with an adversarial verifier or an automatic check confirming each change. All discoveries and ladders went through these rounds. The authors also inspected samples of the labels and audit results by hand.

The table below counts the problems found in the first version of the data and in the final version. A trace leaks hindsight when it mentions "the paper", "the authors" or an arXiv id, since the scientist reasoning at the time could not refer to the finished paper. Ladders chain several methods, so their steps are counted separately from the discoveries.

ProblemWhereFirst versionFinal version
Leaks hindsight from the finished paperDiscoveries82 of 1,1842
Leaks hindsight from the finished paperReasoning of ladder steps48 of 6410
Leaks hindsight from the finished paperAnswers of ladder steps65 of 6410
Answer code is not the original implementationDiscoveries595 of 1,1726
Non-English text in the reasoningDiscoveries9 of 1,1840

The paper also gives examples that can be checked against the sources: Appendix B quotes, for each kind of source, records that hold reasoning absent from the paper, and Figure 5 and Appendix C tie the steps of full traces to the sources they come from. Finally, as an indirect test, rebuilding each trace from the paper alone, where post-hoc rationalization is most likely, lowers every metric, as the ablation table in our first comment shows.

"...It remains to be validated that the resulting gains indicate improved scientific reasoning..."

Thank you for this point. We agree that scores alone cannot show this, so Section 4 tests it in three further ways. The model judges research better although it was never trained to judge: on four tasks from three third-party sources, such as picking the equation the authors wrote or the paper later cited more, RL from the prior is ahead at 4B (Section 4.1). It weighs more methods before committing to one and sets most of them aside (Section 4.2). And human experts find that it reaches simple methods from first principles rather than by retuning known ones (Section 4.3). We have revised the opening of Section 4 to state this purpose more clearly.


Thank you again for your time and effort. We hope our response has addressed your concerns. If the issues have been resolved, we would appreciate it if you could reflect this in your evaluation. We welcome any further comments.

Reviewer_CezWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper claims that LLMs used as research agents recombine and retune existing methods rather than invent new ones. Harnesses can only select from what a frozen model produces, and RL mostly sharpens behaviours the model already has. Hence, the authors argue that the reasoning behind how scientists reach new methods has to be supplied as training data before RL. Papers do not contain this reasoning, so the authors reconstruct it from sources around each selected paper.

The contributions are as follows:

  1. A pipeline that collects publicly available records around a paper (ancestor works, code, third-party explanations and author accounts) and reconstructs how its method was discovered. The resulting dataset, Innovation Prior, contains 1,185 discoveries and 164 research ladders.
  2. Experiments showing that using these reconstructions as training data before RL improves the performance of Qwen3.5-4B and Qwen3.5-9B on multiple benchmarks.

Validation:

  1. Main comparison (Table 1). For Qwen3.5-4B and Qwen3.5-9B, the authors compare four arms (Base, InnoPrior, Base+RL and InnoPrior+RL), with the same GRPO recipe used for both RL arms. They evaluate on FrontierCS Algorithm, FrontierCS Research, ALE-Bench and MLS-Bench, and report that InnoPrior+RL scores highest at both sizes. An 8-gram overlap check finds no verbatim contamination with the benchmarks.
  2. Ablations (Table 2). The authors rebuild the dataset without multiple sources and without fallback patterns, and report that removing either component lowers performance for Qwen3.5-4B.
  3. Test-time scaling (Figure 6). On ALE-Bench Lite with 64 rounds of feedback, the authors report that InnoPrior+RL reaches higher scores for the same compute.
  4. Further analyses (Section 4). The authors report higher accuracy on research judgment tasks and more technique families considered per proposal for the FrontierCS and ALE-Bench benchmarks.
Critical strengths
  1. The paper addresses a valid and real gap: the lack of training data that captures how scientists reach new methods. Papers record finished results, not how they were reached. The authors reconstruct this reasoning from the records around each paper rather than from the paper itself, and Appendices B and C give convincing, verifiable examples of reasoning that is present in these sources but absent from the papers. This matters to my assessment because the dataset is a useful contribution even apart from the RL results.
  2. The construction protocol is rigorous. Contexts are cut off at the discovery year and may not hint at the method, even conceptually, and traces may not report outcomes of experiments the narrator claims to have run (Appendix D). This keeps the traces focused on the discovery process rather than outcomes, and avoids teaching the model to make up evidence in its own reasoning. This matters because it is what makes the traces usable as training data, not just summaries of known results.
Critical weaknesses
  1. The main comparison cannot isolate the effect of the traces (Tables 1-2). InnoPrior+RL differs from Base+RL in three ways at once: a supervised fine-tuning step, the 338 C++ problems, and the discovery traces. Most of the 4B gap remains even when the traces are degraded (FrontierCS Research: 20.47 without multiple sources vs 20.88 full vs 9.07 for Base+RL). The baseline is also unreliable, as 4B Base+RL falls below the untrained Base (FrontierCS Algorithm 4.89 → 0.61; ALE-Bench 200 → 91). Results seem to be reported from a single RL run per arm, with no confidence intervals. This is the main reason for my rating, since the central claim is that the traces themselves give RL a better prior.
  2. The benefit shrinks with scale (Table 1). The InnoPrior+RL vs Base+RL gap falls from +5.5 / +11.8 / +229 at 4B to +0.7 / +2.1 / +1 at 9B (FrontierCS Algorithm, FrontierCS Research, ALE-Bench). This suggests that base model capabilities might account for some of the gap. Two sizes do not establish a trend, but the direction contradicts the conclusion’s case for scaling up.
Questions for authors
  1. Supervised fine-tuning (SFT) control. Can you add an arm with the same SFT recipe on the 338 C++ problems alone, and ideally an answer-only arm trained on (t, C) → A without the reasoning traces, followed by the identical RL?
  2. Seeds and the 4B collapse. Can you report at least three RL seeds per arm with confidence intervals (paired across problems)? Is the drop of 4B Base+RL below the untrained Base reproducible, and do you know what causes it?
  3. Ablations at 9B. At 9B, the gaps are small (ALE-Bench 316 vs 317). Can you run the Table 2 ablations at 9B? Since the ablated datasets were already built, this needs only additional SFT and RL runs, not new data construction.
  4. Hyperparameter selection. How were the 0.1× merge weight and the RL hyperparameters chosen, and were they tuned on a specific arm?
6 comments
AuthorsOpenReview ↗

Response to Reviewer CezW (1/3)

Thank you for your detailed review and constructive suggestions. We especially appreciate that you see the dataset as a useful contribution in its own right, and the construction protocol as what makes the traces usable as training data. We address your concerns below.


Isolating the Effect of the Traces

"...InnoPrior+RL differs from Base+RL in three ways at once: a supervised fine-tuning step, the 338 C++ problems, and the discovery traces..."

"...an arm with the same SFT recipe on the 338 C++ problems alone...followed by the identical RL..."

Thank you for this excellent suggestion. We ran exactly this arm as a new experiment, Maintain-only: the same SFT recipe on only the 338 C++ problems, followed by the identical RL, with the merge weight and evaluation also unchanged. It shares the fine-tuning step and the C++ problems with InnoPrior and differs from it only in the discovery traces, so it separates the three factors you list.

ModelFrontierCS Alg. mean@5FrontierCS Alg. best@5FrontierCS Res. mean@5FrontierCS Res. best@5ALE-Bench mean@5ALE-Bench best@5MLS-Bench
4B Base4.8912.406.5216.732004132.98
4B Maintain-only4.109.509.9919.562063544.83
4B InnoPrior3.979.606.9915.971963886.63
4B Base + RL0.612.949.0718.269111019.28
4B Maintain-only + RL5.5110.5419.7526.8025647519.74
4B InnoPrior + RL6.1311.7720.8828.6132064419.95
9B Base4.4712.0710.3727.6624353411.77
9B Maintain-only4.5212.9510.5230.362044344.98
9B InnoPrior4.5712.279.9027.762325247.17
9B Base + RL7.5613.2616.3330.2331663416.59
9B Maintain-only + RL7.1315.1416.0429.7026855713.80
9B InnoPrior + RL8.2514.8218.4131.3231766019.52

Before RL, Maintain-only is not consistently stronger than Base, so the C++ problems add little by themselves. After RL, InnoPrior + RL is above Maintain-only + RL on every metric at 4B and on all but one at 9B, so the gain comes from the discovery traces. We have added Maintain-only to Table 1 and Sections 3.1 and 3.2 of the revised paper.

"...Most of the 4B gap remains even when the traces are degraded..."

Thank you for the careful reading of Tables 1 and 2. Maintain-only + RL goes through the same fine-tuning and RL and differs from our method only in the discovery traces, so it is the direct reference for judging degraded traces. Against it, the full data are higher on every metric, while each ablation falls below it on several, so degraded traces lose much of the gain.

4B, after RLFrontierCS Alg. mean@5FrontierCS Alg. best@5FrontierCS Res. mean@5FrontierCS Res. best@5ALE-Bench mean@5ALE-Bench best@5MLS-Bench
Maintain-only + RL5.5110.5419.7526.8025647519.74
w/o multiple sources + RL5.2210.4720.4728.1924249413.71
w/o fallback patterns + RL6.0911.1719.6826.1825049714.31
InnoPrior + RL6.1311.7720.8828.6132064419.95

"...an answer-only arm trained on (t, C) → A without the reasoning traces..."

We appreciate this suggestion and considered such an arm carefully. For a reasoning model, however, it would not isolate the effect of the traces. Qwen3.5 is a reasoning model, and our SFT keeps its thinking channel on, so training on (t, C) → A would teach it to answer without reasoning and mix the loss of reasoning with the content of the traces. Our ablations instead keep the reasoning and change only its content, and the table above shows that removing either component lowers every metric.

AuthorsOpenReview ↗

Response to Reviewer CezW (2/3)

The 4B Baseline and Seeds

"...4B Base+RL falls below the untrained Base..."

"...Is the drop of 4B Base+RL below the untrained Base reproducible, and do you know what causes it?..."

Thank you for asking about this; we have investigated it in detail, and the drop is reproducible. RL from the 4B base trains normally up to step 15, as the table below shows, where scored counts rollouts with a nonzero score, Inside counts rollouts that end inside the reasoning before any answer, At limit counts rollouts that reach the length limit, Length is in tokens, and the last column gives the entropy of RL from the prior.

StepRewardScoredInsideLengthAt limitEntropyKLGrad normEntropy, prior
16.123%0%15.5K5%0.740.240.860.75
28.831%0%16.0K4%0.730.250.660.74
37.523%0%19.7K12%0.740.220.370.76
48.930%0%19.2K18%0.650.210.430.63
58.227%1%18.9K21%0.620.190.330.61
66.325%5%22.5K48%0.430.130.460.56
73.916%4%25.4K61%0.360.130.390.50
85.716%7%25.7K70%0.240.080.310.41
96.022%4%22.4K49%0.340.120.250.50
109.326%3%20.4K44%0.350.140.240.46
116.426%2%20.0K40%0.350.130.210.43
125.924%1%19.3K37%0.350.140.240.50
138.532%1%17.0K29%0.350.160.210.37
149.735%1%16.2K28%0.320.170.260.48
158.432%2%16.5K30%0.300.170.270.46

From the step-15 checkpoint we ran steps 16 to 20 three times, and all three runs collapse in the same way, as the table below shows with the median rollout length in characters. Under this recipe, RL from the 4B base cannot be trained stably to step 20, so we use Maintain-only + RL as the reference for the traces. We have added this to Section 3.2 of the revised paper.

RunStepInside reasoningLengthZero rewardMean reward
116 / 17 / 18 / 19 / 202% / 5% / 30% / 58% / 86%35.7K / 33.5K / 26.7K / 18.6K / 9.8K66% / 62% / 60% / 67% / 81%6.9 / 8.9 / 11.2 / 8.3 / 3.4
216 / 17 / 18 / 19 / 202% / 5% / 31% / 68% / 93%35.8K / 34.1K / 29.2K / 18.7K / 9.0K67% / 60% / 56% / 73% / 90%7.5 / 10.8 / 13.1 / 7.7 / 2.4
316 / 17 / 18 / 19 / 202% / 7% / 51% / 62% / 87%38.1K / 33.0K / 27.1K / 17.5K / 9.8K65% / 62% / 71% / 70% / 83%8.2 / 11.9 / 7.5 / 7.5 / 3.8

The collapse is written into the weights rather than produced by the sampler: where the step-20 policy stops, its median probability of ending the response is 0.91, against 3×10⁻¹¹ for the step-15 policy, and with the presence penalty off it still ends 96% of fresh rollouts inside the reasoning. It is not a reward exploit either, since the net GRPO advantage of ending inside the reasoning is negative at every step. Two conditions cause it. Without an overlong penalty, early rollouts that hit the length limit score zero and push the policy toward shorter outputs; and from step 6 on, the entropy of RL from the base stays well below that of RL from the prior, so the policy overshoots until it stops inside the reasoning. From step 15 to step 20, its KL divergence from the reference rises from 0.17 to 0.51 and its gradient norm from 0.27 to 0.86. Under the same recipe, RL from the prior stays stable, with 1% of its rollouts ending inside the reasoning at each of steps 16 to 20.

"...a single RL run per arm, with no confidence intervals..."

"...at least three RL seeds per arm with confidence intervals..."

Thank you for this suggestion; we agree that more runs would strengthen the comparison. Repeating every arm is beyond our compute budget, so we focused the reruns on the baseline in question, shown above. The results are also consistent: each FrontierCS and ALE-Bench score pools five samples per problem, and RL from the prior is above RL from the base on every metric at both sizes.

AuthorsOpenReview ↗

Response to Reviewer CezW (3/3)

Results at 9B

"...The benefit shrinks with scale...the direction contradicts the conclusion's case for scaling up..."

Thank you for pointing this out. For an academic lab, fine-tuning and running RL at both 4B and 9B is already a substantial scale, and larger models are beyond our compute budget. At 9B, the prior is still above RL from the base on every metric, and Figure 6 shows that it keeps the lead with test-time compute while using fewer tokens. The scaling up proposed in our conclusion (Lines 513–520) is to add the prior where it belongs, during the mid-training of a larger model (Lines 256–258). Only a model developer can do this; an academic lab can only approximate it with a small update to released checkpoints, as we do at 4B and 9B. We have revised the conclusion to state this more precisely.

"...Can you run the Table 2 ablations at 9B?..."

Thank you for the suggestion; we agree that 9B ablations would be informative, but their cost is beyond our compute budget. The 4B ablations already answer the question clearly: under the same training budget, each ablation arm falls below our method on every metric, as the ablation table in our first comment shows. At 9B, Maintain-only + RL also isolates the traces as a whole, and the prior is above it on all but one metric.

Hyperparameter Selection

"...How were the 0.1× merge weight and the RL hyperparameters chosen, and were they tuned on a specific arm?..."

Thank you for asking; we agree these choices should be stated explicitly. A prior of this kind belongs in mid-training (Lines 256–258), which an academic lab cannot run. The traces are written by other models and are off-policy for the released checkpoint (Lines 260–262), so fine-tuning it on them in full risks catastrophic forgetting; following common practice in model merging, we merge a tenth of the fine-tuned update into the checkpoint, which adds the prior while avoiding this forgetting. The merge weight is set once and shared by every fine-tuned arm, every RL arm runs the same recipe from an existing configuration, and none of these settings was tuned on any of the compared arms. We have added this to Section 3.1 and Appendix E of the revised paper.


Thank you again for your time and effort. We hope our response has addressed your concerns. If the issues have been resolved, we would appreciate it if you could reflect this in your evaluation. We welcome any further comments.

Reviewer_MDV6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Innovation Prior, a dataset of reconstructed scientific discovery trajectories that capture how researchers develop new methods, including intermediate reasoning and unsuccessful approaches. The authors use this dataset for SFT before RL, aiming to improve models' ability to discover new algorithms and research ideas. Experiments across multiple research benchmarks show that models initialized with Innovation Prior achieve better downstream RL performance than those trained directly from the base model. Additional analyses suggest improvements in exploration diversity, test-time scaling, and research judgment.

Critical strengths
  • This is an important research direction. Improving models' ability to discover new methods rather than merely apply existing ones is an important problem. Training on reconstructed scientific discovery processes is an interesting approach.
  • The empirical results are encouraging. The paper demonstrates improvements across several research-oriented benchmarks, particularly after RL. I also appreciate the evaluations of test-time scaling and exploration behavior.
Critical weaknesses
  • My main concerns is that there is insufficient evidence that the gains are specific to innovation-oriented supervision. The main comparison is against a base model without additional SFT. However, Innovation Prior contains extensive algorithmic knowledge, technical reasoning, and implementation details, all of which could independently improve downstream performance. The existing ablations do not adequately isolate these effects. Without comparisons against matched conventional reasoning or technical SFT, it is unclear whether learning the scientific discovery process is responsible for the observed gains.
  • Relatedly, there is limited evidence for the proposed mechanism. The authors attribute the improvements to a better innovation prior that enables broader exploration. However, the exploration analysis primarily counts mentions of existing technique families, which could also reflect increased technical knowledge or changes in reasoning style. Furthermore, some of the largest gains occur where the Base + RL baseline degrades substantially, making it unclear whether the method improves discovery capabilities or primarily stabilizes RL training.
Questions for authors
  1. Can the authors compare against conventional reasoning SFT constructed from the same scientific discoveries, with matched training budgets? For example, these traces could contain the same technical knowledge, derivations, and implementations without the reconstructed exploration process. This would be particularly important for establishing the contribution of innovation-specific supervision.
  2. Could the authors provide longer RL training curves or additional seeds to clarify whether Innovation Prior improves the eventual performance of RL or primarily prevents degradation of the base model?
  3. Is there stronger evidence that the models learn more effective exploration strategies rather than simply mentioning more known techniques? For example, measuring diversity among implemented solutions and their effectiveness could help distinguish these explanations.
5 comments
AuthorsOpenReview ↗

Response to Reviewer MDV6 (1/3)

Thank you for your thoughtful review, and for recognizing that improving models' ability to discover new methods is an important direction and that our results, test-time scaling and exploration analyses are encouraging. We address your concerns below.


Isolating the Effect of the Discovery Traces

"...Innovation Prior contains extensive algorithmic knowledge, technical reasoning, and implementation details, all of which could independently improve downstream performance. The existing ablations do not adequately isolate these effects..."

Thank you for raising this important point. We ran a new experiment, Maintain-only. It fine-tunes the model on only the 338 C++ problems that Innovation Prior includes to maintain coding ability, without any discovery traces, and then runs the same RL; the SFT recipe, merge weight, RL recipe and evaluation are all unchanged. Maintain-only thus shares the fine-tuning step and the C++ data with InnoPrior and differs from it only in the discovery traces.

ModelFrontierCS Alg. mean@5FrontierCS Alg. best@5FrontierCS Res. mean@5FrontierCS Res. best@5ALE-Bench mean@5ALE-Bench best@5MLS-Bench
4B Base4.8912.406.5216.732004132.98
4B Maintain-only4.109.509.9919.562063544.83
4B InnoPrior3.979.606.9915.971963886.63
4B Base + RL0.612.949.0718.269111019.28
4B Maintain-only + RL5.5110.5419.7526.8025647519.74
4B InnoPrior + RL6.1311.7720.8828.6132064419.95
9B Base4.4712.0710.3727.6624353411.77
9B Maintain-only4.5212.9510.5230.362044344.98
9B InnoPrior4.5712.279.9027.762325247.17
9B Base + RL7.5613.2616.3330.2331663416.59
9B Maintain-only + RL7.1315.1416.0429.7026855713.80
9B InnoPrior + RL8.2514.8218.4131.3231766019.52

Before RL, neither Maintain-only nor InnoPrior is consistently stronger than Base, so neither the C++ problems nor the knowledge in the traces raises scores by itself. After RL, InnoPrior + RL is above Maintain-only + RL on every metric at 4B and on all but one at 9B, so the gain comes from the discovery traces. We have added Maintain-only to Table 1 and Sections 3.1 and 3.2 of the revised paper.

"...compare against conventional reasoning SFT constructed from the same scientific discoveries, with matched training budgets...the same technical knowledge, derivations, and implementations without the reconstructed exploration process..."

Thank you for this concrete proposal. This is exactly our w/o fallback patterns ablation in Table 2. It is built from the same discoveries, keeps the same technical knowledge, derivations and implementations, and removes only the reconstructed exploration, the routes the authors tried and abandoned. It is trained on the same number of examples for the same number of optimizer steps as the full data. As the table below shows, removing the exploration lowers every metric, and so does rebuilding each trace from the paper alone. The same technical knowledge, without the reconstructed discovery process, therefore does not give the gain. We have revised Section 3.3 to state this more clearly.

4B, after RLFrontierCS Alg. mean@5FrontierCS Alg. best@5FrontierCS Res. mean@5FrontierCS Res. best@5ALE-Bench mean@5ALE-Bench best@5MLS-Bench
w/o multiple sources + RL5.2210.4720.4728.1924249413.71
w/o fallback patterns + RL6.0911.1719.6826.1825049714.31
InnoPrior + RL6.1311.7720.8828.6132064419.95
AuthorsOpenReview ↗

Response to Reviewer MDV6 (2/3)

Discovery or RL Stability

"...some of the largest gains occur where the Base + RL baseline degrades substantially, making it unclear whether the method improves discovery capabilities or primarily stabilizes RL training..."

We understand this concern, and we agree that a degrading baseline alone cannot separate discovery from stability, so we also compare with runs that do not degrade. Maintain-only + RL trains stably, and the prior leads it on every metric at 4B; at 9B, RL from the base is stable as well, and the prior is above it on every metric. The prior therefore raises where RL ends, beyond preventing degradation.

"...longer RL training curves or additional seeds to clarify whether Innovation Prior improves the eventual performance of RL or primarily prevents degradation of the base model..."

Thank you for this suggestion. We focused the reruns on RL from the 4B base. It trains normally up to step 15, as the table below shows, where scored counts rollouts with a nonzero score, Inside counts rollouts that end inside the reasoning before any answer, At limit counts rollouts that reach the length limit, Length is in tokens, and the last column gives the entropy of RL from the prior. The last column also shows that the prior keeps the entropy of the policy well above that of RL from the base.

StepRewardScoredInsideLengthAt limitEntropyKLGrad normEntropy, prior
16.123%0%15.5K5%0.740.240.860.75
28.831%0%16.0K4%0.730.250.660.74
37.523%0%19.7K12%0.740.220.370.76
48.930%0%19.2K18%0.650.210.430.63
58.227%1%18.9K21%0.620.190.330.61
66.325%5%22.5K48%0.430.130.460.56
73.916%4%25.4K61%0.360.130.390.50
85.716%7%25.7K70%0.240.080.310.41
96.022%4%22.4K49%0.340.120.250.50
109.326%3%20.4K44%0.350.140.240.46
116.426%2%20.0K40%0.350.130.210.43
125.924%1%19.3K37%0.350.140.240.50
138.532%1%17.0K29%0.350.160.210.37
149.735%1%16.2K28%0.320.170.260.48
158.432%2%16.5K30%0.300.170.270.46

From the step-15 checkpoint we ran steps 16 to 20 three times, and all three runs collapse in the same way, as the table below shows with the median rollout length in characters. Under this recipe, RL from the 4B base cannot be trained stably to step 20, and we have added this to Section 3.2 of the revised paper.

RunStepInside reasoningLengthZero rewardMean reward
116 / 17 / 18 / 19 / 202% / 5% / 30% / 58% / 86%35.7K / 33.5K / 26.7K / 18.6K / 9.8K66% / 62% / 60% / 67% / 81%6.9 / 8.9 / 11.2 / 8.3 / 3.4
216 / 17 / 18 / 19 / 202% / 5% / 31% / 68% / 93%35.8K / 34.1K / 29.2K / 18.7K / 9.0K67% / 60% / 56% / 73% / 90%7.5 / 10.8 / 13.1 / 7.7 / 2.4
316 / 17 / 18 / 19 / 202% / 7% / 51% / 62% / 87%38.1K / 33.0K / 27.1K / 17.5K / 9.8K65% / 62% / 71% / 70% / 83%8.2 / 11.9 / 7.5 / 7.5 / 3.8
AuthorsOpenReview ↗

Response to Reviewer MDV6 (3/3)

Evidence for Broader Exploration

"...the exploration analysis primarily counts mentions of existing technique families, which could also reflect increased technical knowledge or changes in reasoning style..."

Thank you for raising this. We agree that mention counts alone cannot rule this out, and three results speak against it. First, most of the families the prior adds never reach the final code, at both sizes and on all three benchmarks (Lines 411–414); added knowledge or a new style would show up in the code as well, whereas weighing alternatives and then dropping them is exploration, and we have revised Section 4.2 to state this more clearly. Second, during RL the prior keeps the entropy of the policy well above that of RL from the base, as the first table of our previous comment shows, so the model keeps more options open instead of narrowing onto the methods it already favors; we have added this to Section 4.2 of the revised paper. Third, in the case studies, human experts find that the higher score comes from one simple insight rather than from a heavier search over known methods (Section 4.3, Appendix I).

"...stronger evidence that the models learn more effective exploration strategies...measuring diversity among implemented solutions and their effectiveness..."

Thank you for this insightful question and the concrete suggestion. The table below gives the counts behind Figure 8, where an asterisk (*) marks a significant difference in a paired sign test over problems at the 0.05 level. The prior weighs significantly more families than RL from the base at both sizes and on every benchmark, except for five proposals on 9B FrontierCS Research. The breadth also pays off: among the proposals for the same problem, those that weigh more families for each family they implement score higher (Lines 414–416).

Technique families considered by k proposals togetherFrontierCS Alg. k=1FrontierCS Alg. k=5FrontierCS Res. k=1FrontierCS Res. k=5ALE-Bench k=1ALE-Bench k=5
4B Base + RL2.495.854.247.342.526.20
4B InnoPrior + RL6.21 *12.64 *5.49 *9.33 *6.48 *13.80 *
9B Base + RL8.1415.486.9110.888.9717.35
9B InnoPrior + RL9.48 *17.08 *7.46 *11.3011.25 *20.50 *

Thank you again for your time and effort. We hope our response has addressed your concerns. If the issues have been resolved, we would appreciate it if you could reflect this in your evaluation. We welcome any further comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.