acceptodds
Under review as a conference paper at ICLR 2027

REINFORCEMENT LEARNING FROM ENVIRONMENT KNOWLEDGE

Abstract

Reinforcement learning with verifiable rewards (RLVR) has proven effective in training large language models (LLMs) for long-horizon agentic tasks. In specialized domains, however, agents often need to acquire unfamiliar environment knowledge to achieve expert-level performance. For domains where expert demonstrations are scarce but unstructured materials such as documentation are present, RL enables the model to explore on its own and acquire new capabilities through environment interaction and feedback. Yet existing RL recipes mainly optimize for the solution's score, which discourages knowledge acquisition when early attempts to apply unfamiliar techniques lower it. To bridge this gap, we introduce Reinforcement Learning from Environment Knowledge (RLEK), an RL-based method that encourages knowledge acquisition alongside optimizing the solution score. Concretely, we augment vanilla performance episodes with dedicated exploration episodes, each of which starts from a solution the model itself produced earlier in training and is rewarded for successfully adopting a missing documented technique. We study RLEK in GPU kernel generation for a new hardware generation whose instruction set is largely unfamiliar to the model, while relevant documentation is available in the environment. On six held-out GPU kernel tasks, the RLEK-trained model uses documented new techniques in 12% of its passing kernels, compared with none under vanilla RL. RLEK achieves higher best-of-16 performance on five tasks and matches vanilla RL on the sixth, while maintaining similar median performance among correct solutions on most tasks. On Kimi Delta Attention, it achieves the highest median and best-of-16 performance among the evaluated models, exceeding Claude Opus 4.8 under matched context length and time budgets.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Agentic RL, Multi-turn RL, Long-horizon, Kernel Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_c94aOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a challenge in agentic RL: when relevant documentation is available in the environment, optimizing only the final solution’s performance may discourage agents from adopting unfamiliar but potentially valuable techniques whose initial implementations often fail or reduce performance. The authors propose Reinforcement Learning from Environment Knowledge (RLEK), which augments ordinary performance episodes with exploration episodes. These episodes start from correct solutions previously produced by the model and reward the successful adoption of documented techniques absent from the starting solution, provided that the resulting solution is correct and the adopted technique performs actual work. The method is evaluated on Blackwell GPU kernel generation. Compared with vanilla RL, RLEK-107 increases new-technique use among passing kernels on held-out tasks and improves best-of-16 performance on five of the six tasks, matching vanilla RL on the remaining task.

Critical strengths
  1. The paper addresses a practical obstacle in learning complex agentic behaviors. In complex programming tasks, early attempts to apply unfamiliar techniques often reduce performance, making such behavior difficult to reinforce through performance-based rewards alone. This is an important problem in agentic RL.

  2. RLEK separates capability acquisition from performance optimization through a clear and actionable training design. The method demonstrates gains on GPU kernel optimization tasks, and the paper provides behavioral analyses that help characterize the changes induced by training.

Critical weaknesses
  1. The comparison does not sufficiently isolate the contributions of the starting solution, exploration task description, and adoption reward. RLEK changes several factors simultaneously. The vanilla RL control includes only eight seeded tasks that start from frontier kernels, whereas RLEK includes 35 exploration tasks. Although the authors attempt to control for the starting-solution factor, task coverage and the resulting training distributions are still not fully matched.

  2. Improvements in best-of-16 performance come with lower correctness, while median performance remains broadly comparable on most tasks. In Table 3, RLEK-107 achieves higher new-technique use than vanilla RL but lower average correctness. The performance gains are primarily reflected in the best result among 16 episodes per task. Apart from kda, median performance among passing kernels does not show a clear improvement over vanilla RL. This leaves a trade-off between the quality of the best sampled solution and the reliability of generating correct solutions.

Questions for authors

Please address the two concerns raised in Critical Weaknesses. In addition:

  1. The new-technique use metric relies partly on LLM judges. How accurate are these judges in determining whether a technique has been successfully adopted?

  2. Figure 5 shows that RLEK reaches the same mean reward with fewer performance episodes, but exploration episodes are not included on the horizontal axis. Could the authors provide a more detailed accounting of total training costs and the corresponding comparison, including exploration episodes?

2 comments
AuthorsOpenReview ↗

Response to Reviewer c94a

Response to Reviewer c94a

We thank the reviewer for the positive assessment and the constructive questions, which helped us strengthen the paper.

W1. Isolating the exploration episodes. We ran the requested control: vanilla RL on RLEK's exact 78-task set. The 35 additional tasks start from the same kernels k0k_0 as RLEK's exploration tasks but use the performance description and reward; everything else is unchanged.

held-out, median / best (passed of 16)kdadsaglmnsamgqmgrnew-tech. useavg corr.
vanilla RL (105)3.73 / 8.55 (16)2.4 / 3.6 (15)1.7 / 2.4 (15)1.31 / 3.96 (16)8.07 / 8.75 (16)0.63 / 0.76 (12)0%0.94
vanilla RL, RLEK task set (105)7.11 / 8.31 (15)2.5 / 4.3 (16)1.7 / 2.7 (16)1.18 / 3.86 (16)7.99 / 8.62 (16)0.63 / 0.66 (14)1%0.97
RLEK (107)8.95 / 13.64 (15)2.5 / 5.9 (13)1.0 / 2.4 (13)1.27 / 4.13 (15)8.07 / 11.64 (16)0.65 / 1.19 (11)12%0.86

Matching the task set does not move vanilla RL: RLEK's best kernel is still faster on 5 of 6 tasks (1.3x to 1.6x on kda, dsa, mgq, mgr), and vanilla RL still only gives 1% new-technique use.

W2. Best-of-16 vs. correctness and median.

avg corr.best-to-median speed ratio (median over tasks)
vanilla RL (105)0.941.4x
vanilla RL, RLEK task set (105)0.971.4x
RLEK (107)0.862.1x
RLEK (180)0.921.6x

The lower correctness reflects more attempts at new techniques, as for DSV4-Flash in Table 2: not every attempt lands, which lowers the average correctness but raises the best-to-median ratio. In kernel generation, many kernels are usually sampled to get some good candidates; for example, many existing pipelines sample and search over many candidates per task (e.g., AlphaEvolve [1]). We therefore report best-of-16 as the main metric, alongside the median and pass rate in Tables 2 and 3.

[1] Novikov et al. AlphaEvolve: A coding agent for scientific and algorithmic discovery. arXiv:2506.13131, 2025.

Q1. Judge accuracy. We randomly sampled 50 adoption decisions from the solution database (25 granted, 25 denied) and compared them with human judgment: 0 false positives and 0 false negatives. We will add this check as an appendix.

Q2. Total training cost in Figure 5. Figure 5 already marks the same trainer step for both runs (hollow markers at step 100), i.e. the same total training cost with exploration episodes included. At that step RLEK's mean performance reward (≈0.035) is slightly above vanilla RL's (≈0.033), although only ≈55% of its episodes are performance episodes. We will add a Figure 5 panel against total episodes.

Thanks for your response. I will keep my positive score.

Reviewer_7G3YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how an RL agent can acquire unfamiliar techniques from environment documentation when expert demonstrations are scarce and vanilla RL tends to reinforce familiar strategies. It proposes RLEK, which augments standard performance episodes with exploration episodes and reward the verified adoption of missing documented techniques. On Blackwell GPU kernel optimization, RLEK increases new-technique use over vanilla RL and achieves higher best-of-16 performance on five of six held-out tasks while matching it on the sixth.

Critical strengths
  1. RLEK introduces a and meaningful mechanism that allows the agent to practice unfamiliar documented techniques using environment feedback. This provides a potentially useful alternative to pipelines that depend on expert demonstrations such as SFT+RL.
Critical weaknesses
  1. Overall, the writing is not clear enough and hard to follow. The method presentation does not clearly distinguish kernel problems, task configurations, episodes, environment turns, and trainer steps. In particular, the paper describes 43 performance tasks and 35 exploration tasks without clearly explaining their sampling schedule or resulting episode ratio. Section 3.3 further interleaves the starting-state construction, reward definition, and online data-generation procedure, making the exact training recipe difficult to follow and reproduce.

  2. RLEK relies on substantially more structured supervision than its framing suggests. Its adoption reward requires an explicit technique set K(D), technique-specific source checks, and proprietary LLM judges, but the paper does not explain how these components could be constructed reliably from arbitrary unstructured materials. Moreover, the technique-frontier definition assumes that established techniques can be accumulated in a single solution. If multiple effective techniques are mutually incompatible, all may become established while no solution contains all of them, making FqF_q​ empty. These assumptions significantly limit the generality and reproducibility of RLEK.

Questions for authors
  1. Could the authors report the total number of performance and exploration episodes, sampled tokens, and compute used by each run? Does the claim of “half the performance episodes” still hold when exploration episodes are included in the training cost?
1 comment

Response to Reviewer 7G3Y

We thank the reviewer for recognizing RLEK as a potentially useful alternative to SFT+RL. We position RLEK as an early exploration of RL-only acquisition of unfamiliar techniques without expert demonstrations. Our experiments on the challenging testbed of GPU kernel generation provide promising initial results, while important limitations and open questions remain. Below we address the reviewer's key concerns.

W1. Terms.

termdefinition
problem qqkernel spec + PyTorch reference + tests
task (q,y)(q, y)problem + starting kernel + type: performance (y=0y{=}0, starts from the naive seed) or exploration (y=1y{=}1, starts from k0(q)∈Fqk_0(q) \in \mathcal{F}_q)
episode τ\tauone agent attempt at a task
turnone reasoning sequence + one shell command and its output
trainer stepone GRPO update on 4 groups of G=16G{=}16 episodes

Sampling schedule. Rollout workers start groups of 16 episodes by cycling through the task list (Algorithm 1, line 3); 43 of RLEK's 78 rows are performance tasks. Rollout is fully asynchronous: each trainer step takes the first 4 groups that finish. The resulting episode counts are in Q1.

Reproduction. We will open-source the full training infrastructure, including the rollout server, reward server, and sandbox server, as well as the tasks, documentation, judge prompts, and checkpoints, for easy reproduction.

W2. Structured supervision and generality. RLEK does not rely on much structured manual supervision. In an exploration task, the model reads the documentation and the starting kernel and decides on its own which documented technique to try. K(D)\mathcal{K}(D) and the related notation only serve to describe the method formally, and they do not imply human curation. Point by point:

  • Technique set K(D)\mathcal{K}(D). Not manually curated. The judge prompt gives a few example techniques, but the judges compare the final solution with the starting kernel in light of the documentation and also identify techniques outside the examples.
  • Technique-specific source checks. We apologize for the ambiguity in Section 4.1. The rule-based source checks mentioned there are not actually needed for adoption verification. Adoption is verified by two LLM judges with one technique-agnostic prompt that checks whether the final solution adopts a documented technique absent from the starting kernel and whether it carries the solution's work. Thus, RLEK does not require human-crafted technique-specific checks for reward assignment. We will clarify this distinction in the revised manuscript.
  • Proprietary LLM judges. Open-weight models give judgments similar to those of proprietary models. We sampled 50 adoption cases from the solution database (25 granted, 25 denied), labeled them by hand, and ran each judge pair 3 times with the same prompt:
judge pairmean agreement with manual labels
GPT-5.6 + Claude Opus 5 (used in our training)94% (46 to 48 of 50)
DeepSeek-V4.1-Flash ×2 (open weights)93% (46 to 48 of 50)

W2b. Incompatible techniques, empty frontier. We thank the reviewer for pointing out this problem in general settings. Although it did not happen in our setting, for domains where techniques truly conflict the frontier can keep the solutions with inclusion-maximal sets of established techniques:

Fq={k∈Bq0:∄k′∈Bq0, Eq∩d(k)⊊Eq∩d(k′)}\mathcal{F}_q = \lbrace k \in \mathcal{B}^0_q : \nexists k' \in \mathcal{B}^0_q, ~\mathcal{E}_q \cap d(k) \subsetneq \mathcal{E}_q \cap d(k') \rbrace

For a nonempty solution buffer, this frontier is guaranteed to be nonempty and reduces to our original definition when a solution contains all established techniques. We will update this in Section 3.3.

Q1. Episodes, tokens, compute.

runperformance episodesexploration episodesgenerated tokensepisodes until unseeded task reward reaches 0.035
vanilla RL (105)6,78401.10B5,184
RLEK (107)2,7204,1921.05B4,864 (2,096 performance)
vanilla RL, RLEK task set (107)6,88001.01Bnot reached

The third run is a new baseline that matches RLEK's starting-kernel task coverage but uses performance descriptions and rewards throughout, allowing us to test whether additional starting-kernel coverage alone explains RLEK's gains.

When exploration episodes are included, RLEK reaches the same performance reward after approximately the same total number of episodes as vanilla RL (4,864 vs. 5,184), but with only 2,096 performance episodes (40% of vanilla RL). We will update figure5 accordingly to indicate the matched-compute results clearly.

We appreciate the reviewer's constructive feedback, which has helped us improve the paper and the presentation. We hope the new experimental evidence and clarifications address your main concerns.

Reviewer_BucsOpenReview ↗
Rating
1
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes Reinforcement Learning from Environment Knowledge (RLEK), a framework for tool-call RL training for GPU kernel optimization. RLEK trains a model to read from documents and produce GPU kernel code. The main motivation is that the document might contain new information on new iterations of GPU generation. In the paper's experimental setup, the authors included two types of tasks / rewards in the training mix, (1) "performance" tasks, where model is expected to rewrite a kernel and be rewarded based on the speed improvements, and (2) "exploration" tasks, where the model is expected to reproduce a documented technique and implement it in a kernel. The author show model trained with RLEK shows better pass rates on 6 held-out tasks compared to training on "performance" tasks alone under similar level of compute budget

Critical strengths
  • GPU Kernel optimization is an important problem to work on.
  • Having OSS async RL recipe for kernel optimization or coding agents in general is good for the research community.
Critical weaknesses

In the current form, the contribution(s) of the paper is framed and positioned disproportionately to the actual work & experiments done, e.g. compare to "vanilla RL", the RLEK only adds a so-called "exploration" task, which is a agentic tool-call task which reads documents about kernel design and implement it. And the authors show that compared to training only on "performance" tasks, i.e. kernel design / rewrite tasks, training on both tasks together would yield performance improvements.

Here are the suggestions I have for making the presentation + experiment setup of the paper more solid.

  • Be precise about the contribution with respect to existing work in the field: e.g. the paper claim RLEK to be a new RL method that "treats acquiring environment knowledge as part of the learning objective alongside optimizing the solution score, whereas under vanilla RL familiar strategies take priority over such acquisition. It lets the policy acquire documented techniques from the environment while optimizing the solution score". For this, there are many existing work for this problem. e.g. for coding agents we have work like Tmax [1] that encapsulates the "reading from environment", and the benefit of "learning to read from environment" was justified by many work outside of coding agents' specific setting, e.g. search R1 [2]. I would suggest authors offer a critical comparison or discussion so that readers know what the precise contribution of the paper is on top of what people already did.

  • Build the experiments around the research hypothesis you want to test: e.g. I see the value add of the current experiments to be "understanding what it takes for GPU kernel optimization RL training work". And I see and would like to believe there are good amount of solid work that goes into making the Async RL + Environments for GPU kernel complication / testing pipeline to work. But since all of the experiments focus around the training mix comparison (e.g. performance + exploration vs. performance only), we don't get to validate the value of other parts of the contribution you did.

[1] Ivison, H., Yin, J.O., Shao, R., Xiao, T., Lambert, N. and Hajishirzi, H., 2026. Tmax: A simple recipe for terminal agents. arXiv preprint arXiv:2606.23321.

[2] Jin, B., Zeng, H., Yue, Z., Yoon, J., Arik, S., Wang, D., Zamani, H. and Han, J., 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. In COLM 2025.

Questions for authors

See above

1 comment

Dear Reviewer Bucs,

Thank you for your time and review! However, we believe there is a fundamental misunderstanding of our research question.

Our work is not about teaching agents to read documentation or retrieve information from the environment. We assume that the base model already possesses these agentic capabilities, such as terminal tool use and information retrieval. In fact, our paper explicitly demonstrates that our selected base model can already independently identify and read relevant documentation before our RL training, yet still fails to successfully apply the documented techniques. Instead, we study a different challenge in agentic RL: early attempts to apply unfamiliar techniques often fail or reduce performance, making such behaviors difficult to reinforce through conventional performance-based rewards.

Therefore, we do not see how TMax or Search-R1 directly addresses our research question. These works investigate developing agentic capabilities such as terminal tool use and information retrieval, whereas we investigate how agents can acquire unfamiliar techniques when these agentic capabilities are already present. We believe work such as TailRL [1], which investigates how conventional RL can lose coverage of rare but valuable behaviors, is more closely related to our research motivation. We would appreciate it if the reviewer could reconsider and update their review to focus on our actual research question.

[1] Ramasubramanian, S., et al. (2026). Tail-Likelihood Reinforcement Learning. arXiv:2609.02987. https://arxiv.org/abs/2609.02987

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.