- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes REI, which learns to generate engineering improvement proposals from historical repository changes and their outcomes. The authors find that reinforcement learning can make the model favor a small number of technical directions, even when its wording remains diverse. To address this, SPOD combines method-category annotation, category-level reward adjustment, and exemption of each category's highest-reward candidate to reduce repeated directions. Experiments cover an industrial recommender system, nanoGPT, and rustc-perf, evaluating proposal success, method coverage, and multi-round improvement.
Critical strengths
-
The problem has practical value. Generating suggestions is inexpensive, but validating them often requires code changes, training, and testing. Learning which directions are worth trying from past engineering experience could reduce unproductive exploration. Section 3.1 presents a clear learning pipeline that connects historical changes with their outcomes.
-
The diversity analysis provides a useful observation. Figures 2–4 show that models can generate differently worded suggestions that follow similar technical directions. Text diversity alone may therefore fail to reveal a narrowing search space. This observation is useful for understanding how ideation models behave during training.
-
The experiments support improved coverage of candidate directions. Tables 2–4 show that SPOD generally covers more relevant method families and reduces concentration on a single family. The paper also evaluates cross-task transfer, and the ablation in Table 5 further supports the roles of its three components. Multiple-seed results and statistical tests on the public benchmark strengthen these findings, while Table 1 also includes actual implementation and training.
Critical weaknesses
-
The experiments establish coverage of relevant directions more directly than the effectiveness of concrete proposals. The offline metrics in Section 4.1 mainly check whether a suggestion belongs to a correct method family. For example, if the historically effective change involves an optimizer, a generated optimizer-related suggestion may receive credit in the category metrics. However, these metrics do not establish whether the specific modification is reasonable or would improve performance. The public multi-round experiment in Section 4.3 also assigns historical gains based on the selected category, rather than executing the model's new suggestions. These results therefore support broader exploration, including better coverage of relevant directions across tasks, but provide less direct evidence that the concrete proposals are more effective.
-
The industrial execution results lack evaluation counts and uncertainty estimates. Table 1 reports encouraging results for both REI and REI+SPOD, but does not state how many tasks were tested or provide uncertainty estimates for the success rates. This makes it difficult to assess how reliable the differences between methods are. In particular, adding SPOD raises Success@1 from 68% to 69% and Success@3 from 79% to 83%; these gains need to be interpreted in light of the evaluation size. The paper already provides multiple-seed results and statistical analysis for the public benchmark, but these do not establish the uncertainty of the industrial execution success rates in Table 1.
Questions for authors
-
How does the method distinguish category bias from genuine effectiveness? Section 3.3 reduces the rewards of some candidates from categories with higher average rewards. However, a category may score highly because it is genuinely more effective for the current task. How do the authors interpret and handle this case? This would affect my understanding of SPOD's reward adjustment and when it is appropriate.
-
How reliable are the industrial execution results? How many test tasks are included in Table 1? Are REI and REI+SPOD compared on the same tasks? Could the authors provide uncertainty estimates, such as confidence intervals, for the success rates or their differences? These details would help assess the reliability of the reported practical gains.
The paper could also briefly discuss similar concurrent work. For example, ResearchTrails also extracts research-decision trajectories from Git commit histories to guide subsequent exploration. A short discussion of the connections and differences in learning goals, supervision signals, and method design would help readers understand REI's contribution. This is a suggestion for the discussion, not an independent negative assessment.