- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies a challenge in agentic RL: when relevant documentation is available in the environment, optimizing only the final solution’s performance may discourage agents from adopting unfamiliar but potentially valuable techniques whose initial implementations often fail or reduce performance. The authors propose Reinforcement Learning from Environment Knowledge (RLEK), which augments ordinary performance episodes with exploration episodes. These episodes start from correct solutions previously produced by the model and reward the successful adoption of documented techniques absent from the starting solution, provided that the resulting solution is correct and the adopted technique performs actual work. The method is evaluated on Blackwell GPU kernel generation. Compared with vanilla RL, RLEK-107 increases new-technique use among passing kernels on held-out tasks and improves best-of-16 performance on five of the six tasks, matching vanilla RL on the remaining task.
Critical strengths
-
The paper addresses a practical obstacle in learning complex agentic behaviors. In complex programming tasks, early attempts to apply unfamiliar techniques often reduce performance, making such behavior difficult to reinforce through performance-based rewards alone. This is an important problem in agentic RL.
-
RLEK separates capability acquisition from performance optimization through a clear and actionable training design. The method demonstrates gains on GPU kernel optimization tasks, and the paper provides behavioral analyses that help characterize the changes induced by training.
Critical weaknesses
-
The comparison does not sufficiently isolate the contributions of the starting solution, exploration task description, and adoption reward. RLEK changes several factors simultaneously. The vanilla RL control includes only eight seeded tasks that start from frontier kernels, whereas RLEK includes 35 exploration tasks. Although the authors attempt to control for the starting-solution factor, task coverage and the resulting training distributions are still not fully matched.
-
Improvements in best-of-16 performance come with lower correctness, while median performance remains broadly comparable on most tasks. In Table 3, RLEK-107 achieves higher new-technique use than vanilla RL but lower average correctness. The performance gains are primarily reflected in the best result among 16 episodes per task. Apart from kda, median performance among passing kernels does not show a clear improvement over vanilla RL. This leaves a trade-off between the quality of the best sampled solution and the reliability of generating correct solutions.
Questions for authors
Please address the two concerns raised in Critical Weaknesses. In addition:
-
The new-technique use metric relies partly on LLM judges. How accurate are these judges in determining whether a technique has been successfully adopted?
-
Figure 5 shows that RLEK reaches the same mean reward with fewer performance episodes, but exploration episodes are not included on the horizontal axis. Could the authors provide a more detailed accounting of total training costs and the corresponding comparison, including exploration episodes?
2 comments
Thanks for your response. I will keep my positive score.
Response to Reviewer c94a
Response to Reviewer c94a
We thank the reviewer for the positive assessment and the constructive questions, which helped us strengthen the paper.
W1. Isolating the exploration episodes. We ran the requested control: vanilla RL on RLEK's exact 78-task set. The 35 additional tasks start from the same kernels k0 as RLEK's exploration tasks but use the performance description and reward; everything else is unchanged.
Matching the task set does not move vanilla RL: RLEK's best kernel is still faster on 5 of 6 tasks (1.3x to 1.6x on kda, dsa, mgq, mgr), and vanilla RL still only gives 1% new-technique use.
W2. Best-of-16 vs. correctness and median.
The lower correctness reflects more attempts at new techniques, as for DSV4-Flash in Table 2: not every attempt lands, which lowers the average correctness but raises the best-to-median ratio. In kernel generation, many kernels are usually sampled to get some good candidates; for example, many existing pipelines sample and search over many candidates per task (e.g., AlphaEvolve [1]). We therefore report best-of-16 as the main metric, alongside the median and pass rate in Tables 2 and 3.
[1] Novikov et al. AlphaEvolve: A coding agent for scientific and algorithmic discovery. arXiv:2506.13131, 2025.
Q1. Judge accuracy. We randomly sampled 50 adoption decisions from the solution database (25 granted, 25 denied) and compared them with human judgment: 0 false positives and 0 false negatives. We will add this check as an appendix.
Q2. Total training cost in Figure 5. Figure 5 already marks the same trainer step for both runs (hollow markers at step 100), i.e. the same total training cost with exploration episodes included. At that step RLEK's mean performance reward (≈0.035) is slightly above vanilla RL's (≈0.033), although only ≈55% of its episodes are performance episodes. We will add a Figure 5 panel against total episodes.