acceptodds
Under review as a conference paper at ICLR 2027

REI: Learning Ideation from Repository Evolution

Abstract

As AI systems become increasingly capable of executing and evaluating experiments, ideation is emerging as a bottleneck in automated scientific and engineering iteration. In many real-world domains, however, useful ideas are sparse and their validation is costly, making direct online exploration inefficient. We observe that repository evolution provides a scalable source of reusable technical experience: code changes record concrete technical hypotheses, while subsequent revisions, adoption, and reuse provide signals about their utility. Based on this observation, we propose REI, a Repository Evolution-driven Ideation framework that learns reusable ideation policies from repository evolution, injecting domain-specific technical experience into model parameters and reducing reliance on costly trial-and-error. We further identify a hidden failure mode in quality-driven ideation optimization: semantic diversity collapse. Although generated ideas can remain lexically diverse, they increasingly concentrate in a small number of dominant method families, limiting cross-task generalization and long-tail exploration. To address this problem, we introduce Semantic Prior Optimization and Debiasing (SPOD), a diversity-preserving alignment mechanism that makes method identity explicit, corrects category-level reward bias, and limits redundant accumulation within the same method family while retaining strong representatives. Experiments on public benchmarks and a real-world industrial setting demonstrate improvements in ideation quality, method diversity, cross-task generalization, and long-tail exploration.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Recommender System, Diversity Generation, AI4AI

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_aznFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes REI, which learns to generate engineering improvement proposals from historical repository changes and their outcomes. The authors find that reinforcement learning can make the model favor a small number of technical directions, even when its wording remains diverse. To address this, SPOD combines method-category annotation, category-level reward adjustment, and exemption of each category's highest-reward candidate to reduce repeated directions. Experiments cover an industrial recommender system, nanoGPT, and rustc-perf, evaluating proposal success, method coverage, and multi-round improvement.

Critical strengths
  1. The problem has practical value. Generating suggestions is inexpensive, but validating them often requires code changes, training, and testing. Learning which directions are worth trying from past engineering experience could reduce unproductive exploration. Section 3.1 presents a clear learning pipeline that connects historical changes with their outcomes.

  2. The diversity analysis provides a useful observation. Figures 2–4 show that models can generate differently worded suggestions that follow similar technical directions. Text diversity alone may therefore fail to reveal a narrowing search space. This observation is useful for understanding how ideation models behave during training.

  3. The experiments support improved coverage of candidate directions. Tables 2–4 show that SPOD generally covers more relevant method families and reduces concentration on a single family. The paper also evaluates cross-task transfer, and the ablation in Table 5 further supports the roles of its three components. Multiple-seed results and statistical tests on the public benchmark strengthen these findings, while Table 1 also includes actual implementation and training.

Critical weaknesses
  1. The experiments establish coverage of relevant directions more directly than the effectiveness of concrete proposals. The offline metrics in Section 4.1 mainly check whether a suggestion belongs to a correct method family. For example, if the historically effective change involves an optimizer, a generated optimizer-related suggestion may receive credit in the category metrics. However, these metrics do not establish whether the specific modification is reasonable or would improve performance. The public multi-round experiment in Section 4.3 also assigns historical gains based on the selected category, rather than executing the model's new suggestions. These results therefore support broader exploration, including better coverage of relevant directions across tasks, but provide less direct evidence that the concrete proposals are more effective.

  2. The industrial execution results lack evaluation counts and uncertainty estimates. Table 1 reports encouraging results for both REI and REI+SPOD, but does not state how many tasks were tested or provide uncertainty estimates for the success rates. This makes it difficult to assess how reliable the differences between methods are. In particular, adding SPOD raises Success@1 from 68% to 69% and Success@3 from 79% to 83%; these gains need to be interpreted in light of the evaluation size. The paper already provides multiple-seed results and statistical analysis for the public benchmark, but these do not establish the uncertainty of the industrial execution success rates in Table 1.

Questions for authors
  1. How does the method distinguish category bias from genuine effectiveness? Section 3.3 reduces the rewards of some candidates from categories with higher average rewards. However, a category may score highly because it is genuinely more effective for the current task. How do the authors interpret and handle this case? This would affect my understanding of SPOD's reward adjustment and when it is appropriate.

  2. How reliable are the industrial execution results? How many test tasks are included in Table 1? Are REI and REI+SPOD compared on the same tasks? Could the authors provide uncertainty estimates, such as confidence intervals, for the success rates or their differences? These details would help assess the reliability of the reported practical gains.

The paper could also briefly discuss similar concurrent work. For example, ResearchTrails also extracts research-decision trajectories from Git commit histories to guide subsequent exploration. A short discussion of the connections and differences in learning goals, supervision signals, and method design would help readers understand REI's contribution. This is a suggestion for the discussion, not an independent negative assessment.

Reviewer_8RvEOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to learn an ideation policy from repository history. Each sample has a context, a code change, and its recorded outcome. REI uses SFT on successful changes, trains a binary reward model on successful and failed changes, and then uses GRPO. The paper finds that GRPO makes the ideas collapse into a few method families. To reduce this collapse, SPOD subtracts a category reward prior from each reward and keeps the best idea in each category unchanged. On an industrial recommender system, nanoGPT and rustc-perf, REI with SPOD improves category-level quality and diversity over GRPO.

Critical strengths

Overall, I give this paper interesting. The paper solves a problem in a specific scenario. This scenario has a large amount of past experience. It is not possible to put all of this experience into the context. Instead, the paper uses training to put this experience into the model parameters. The paper uses this specialized model to control a general LLM. I find this idea very interesting.

Critical weaknesses
  1. The method needs a large number of past experiments with recorded results in the target domain.

  2. Each domain needs its own trained model. The model can only propose improvements for the current problem. Its general ability is low. The cause is that the authors fine-tune the idea policy on one specific domain. For example, if you train the policy for the nanoGPT problem, it cannot propose better ideas in a different domain.

Questions for authors

Q1. How do you design the reward score? The paper trains the reward model as a binary classifier with 0/1 labels. How do you get a continuous reward score from this model? Please explain this clearly.

Q2. How can we trust that the reward model is accurate? The reward model gives a score to each idea, but the method does not run the ideas. Is this score reliable? I also have a suggestion. If you replace the policy or the reward model with a general LLM API, is the result stronger? For example, a general LLM proposes the ideas, and the domain reward model only ranks them.

Reviewer_muUnOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper collects rollouts during automated research, and use those data to train idea generators and idea rankers. During training, they further tweak the reward to penalize the idea ranker's biases towards certain type of ideas, hence improving the diversity of explored ideas.

Critical strengths

The formulation of proposing ideas based on a codebase is reasonable. Compared to proposing ideas with only the problem description, this formulation aligns with many real-world use cases.

Critical weaknesses
  1. Overall, the result presentation is quite unclear and misleading. The actual results often cannot support the claims made in the paper. For example:
  • One major claim is that "stronger verifiers accelerate diversity collapse", as shown in Figure 4. First, the main paper does not clarify how are these different verifiers created. I check the appendix and find they are just implementedy by adding random noise to a single verifier's predictions. That said, Figure 4 only supports that "if a verifier has substantial bias towards certain kinds of ideas, trusting it more would lead to lower idea diversity". But there are definitely ways that we can mitigate biases in verifiers, so stronger verifiers lead to more diverse ideas.
  • In Table 1, the paper seems to show that their small 8B models outperform some large close-weight models. But it's important to clarify that these 8B models are exactly trained on the same distribution of problems. So they can get good scores even just simply by memorizing good ideas in their training data.
  1. Lack important simple baselines for SPOD. Instead of tuning rewards and running RL, there are in fact many simple baselines that the paper miss, such as tuning the training data mix of the verifier, do some simple inference-time invertention on the verifier.
Questions for authors

see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.