- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies generalization in terminal agents trained with reinforcement learning (RL) in synthetic environments. It proposes the Agentic Compositional Generalization (ACG) hypothesis, which posits that pre-training and supervised fine-tuning (SFT) mainly provide low-level skills, whereas RL primarily improves how these skills are selected, composed, and coordinated across multi-turn interactions. The paper also introduces RIVER, which improves RL training signals through environment quality auditing, oracle-agent solvability filtering, and a repetition penalty. The approach is evaluated across multiple model scales, agent harnesses, and terminal-agent benchmarks. Skill-structure analysis, behavior–success prediction, SFT data ablations, and cross-domain RL experiments provide empirical support for the ACG hypothesis.
Critical strengths
-
The training recipe is practically valuable and is supported by a relatively broad empirical evaluation. RIVER achieves positive results across multiple model scales, agent harnesses, and RL settings, providing useful evidence that improving environment quality and reducing low-value training data can benefit terminal-agent RL.
-
The analysis offers an insightful and testable perspective. By combining skill-structure analysis, behavior prediction, SFT initialization studies, and cross-domain experiments, the paper provides a useful way to study what RL changes in terminal agents and demonstrates awareness of the distinction between correlational evidence and the effects of training interventions.
Critical weaknesses
-
The mechanistic analysis is still largely based on indirect evidence. Chapter 4 provides relatively consistent support for the ACG hypothesis through skill-structure analysis, behavior features, and cross-domain experiments. However, the current definitions of “skills” and “behaviors” are primarily based on trajectory-level statistical features. These results are therefore best interpreted as empirical evidence consistent with the ACG hypothesis. A further discussion of the scope and limitations of these measurements would make the mechanistic interpretation more rigorous.
-
The contributions of the individual RIVER components could be disentangled more clearly. The paper already includes a random-environment control and behavior-verifier ablations, making the overall experimental evaluation fairly comprehensive. However, RIVER combines environment filtering, pass@2 filtering, and a repetition penalty. More information would help clarify the relative contribution of each component, as well as how the difficulty and domain distribution of the environments change after filtering.
Questions for authors
-
In Section 4.1, the skill labels primarily indicate whether a certain type of operation appears in a trajectory. Could the authors clarify whether the correctness of the resulting operation is also taken into account, and how the reliability of the GPT-5.4 skill annotations is validated?
-
After rubric-based filtering and pass@2 filtering, did the difficulty and domain distribution of the environments change substantially? The appendix already provides some related statistics, but a further description of the overall task-distribution changes before and after filtering would help readers better understand whether the performance gains mainly come from improved reward integrity or from changes in the task set.
-
Section 4.4 shows that RL training on a subset of domains can improve performance on other domains. Could the authors explain the rationale for selecting these domain subsets, and clarify whether the same trend holds for different domain combinations with the same number of training domains?
4 comments
Response [2/3]
Q1. Skill correctness and annotation reliability
In Section 4.1, the skill labels primarily indicate whether a certain type of operation appears in a trajectory. Could the authors clarify whether the correctness of the resulting operation is also taken into account, and how the reliability of the GPT-5.4 skill annotations is validated?
Clarification of skill labels
We would like to clarify that, in the experiments supporting our ACG hypothesis in Section 4.1, skill-presence labels indicate that the corresponding skill is performed correctly. Also as defined on line 103, a skill is the ability to execute actions correctly.
Cross-model validation of the skill annotations
We used Claude Opus 4.6 to evaluate the GPT-5.4 skill annotations in 100 randomly sampled trajectories. The judgments were categorized as follows:
- Supported: The annotation is judged correct and supported by direct evidence.
- Unsupported: The annotation lacks direct supporting evidence.
- Uncertain: The judge model cannot reach a clear determination.
The results are shown below. In this cross-model audit, 94.5% of the reviewed annotations were judged to be supported, providing additional model-based evidence for their reliability.
| Judgment | Percentage |
|---|---|
| Supported | 94.5% |
| Unsupported | 4.1% |
| Uncertain | 1.4% |
Q2. Changes in task difficulty and domain distribution
After rubric-based filtering and pass@2 filtering, did the difficulty and domain distribution of the environments change substantially? The appendix already provides some related statistics, but a further description of the overall task-distribution changes before and after filtering would help readers better understand whether the performance gains mainly come from improved reward integrity or from changes in the task set.
The task-category distributions of the original TMax collection and its filtered subsets are shown in the following table. Qualitatively, after pass@2 filtering, the difficulty decreases slightly because hard problems may also be filtered out.
| Original TMax category | Before filtering | After rubric filtering | After rubric + pass@2 |
|---|---|---|---|
| Data querying | 12.0% | 13.9% | 15.2% |
| Data science | 11.8% | 13.1% | 12.7% |
| Scientific computing | 11.7% | 11.0% | 8.9% |
| Software engineering | 11.5% | 7.2% | 6.5% |
| Debugging | 11.4% | 10.2% | 11.1% |
| File operations | 11.2% | 12.8% | 13.4% |
| Data processing | 10.8% | 15.6% | 17.0% |
| Security | 10.1% | 9.8% | 8.9% |
| System administration | 9.5% | 6.4% | 6.4% |
The task-category distribution remains relatively stable after filtering.
In addition, in Section 4.4, Table 2, and Figure 7, we conducted controlled experiments by training on environments from different domains. We observed that, with environments containing reliable verifiers, the learned behaviors can generalize to different domains, including those excluded from training, and performance in those domains can also improve. These findings provide complementary evidence that the learned behaviors can transfer beyond the RL training domains.
Response [3/3]
Q3. Selection of training-domain subsets
Section 4.4 shows that RL training on a subset of domains can improve performance on other domains. Could the authors explain the rationale for selecting these domain subsets, and clarify whether the same trend holds for different domain combinations with the same number of training domains?
Domain partition
Our domain partition is based on tasks' core problem-solving requirements, with the aim of distinguishing domains by their substantive skill demands. Specifically, we identify skills using task instructions, skill metadata, and the code contained in successful reference trajectories, rather than relying on generic operations shared across domains, such as reading files or running Python. For example, tasks implemented in Python are distinguished according to whether their central computation involves graph algorithms, numerical computation, or data processing.
We consolidate the identified skills into eight domains and assign each task to the domain of its primary skill. This design emphasizes differences in the problems being solved and the core capabilities required, thereby promoting separation between domains in their primary skill requirements.
Evaluated training-domain subsets
RL (2 domains) is trained on Debug and Systems, whereas RL (4 domains) is trained on Data, Graph, Security, and Text. The two training-domain sets are entirely disjoint. As shown in the table below, both models outperform SFT across all eight evaluation domains, including those excluded from their respective RL training sets.
We selected the training-domain subsets based on a qualitative assessment of their primary task and skill requirements. Specifically, we chose Debug and Systems for the two-domain setting, and Data, Graph, Security, and Text for the four-domain setting, forming two disjoint training-domain subsets. The selection was guided by the aim of examining transfer across domains with relatively distinct core problem-solving requirements. Here, domain separation refers to differences in primary skill requirements; the domains may still share general operations and some underlying skills. The results are summarized in the following table.
| Model | Build | Data | Debug | Graph | Numerical | Security | Systems | Text | Overall |
|---|---|---|---|---|---|---|---|---|---|
| RL on (Data, Graph, Security, Text) | 50.8 | 53.8 | 47.8 | 68.8 | 69.7 | 40.6 | 51.2 | 55.6 | 53.6 |
| RL on (Debug, Systems) | 45.9 | 47.7 | 52.2 | 53.1 | 57.6 | 46.9 | 46.4 | 55.6 | 49.2 |
| SFT | 34.4 | 29.2 | 43.5 | 28.1 | 45.5 | 21.9 | 26.2 | 44.4 | 32.3 |
The results show that the same trend of generalizable performance across different domains holds for different domain combinations.
Thanks for your response. I will keep my positive score.
Response [1/3]
We sincerely thank the reviewer for their time and effort in reviewing this paper. We would like to address their concerns and answer their questions in the following content.
W1. Scope and limitations of the mechanistic interpretation
One limitation of our current analysis is that we cannot enumerate all skills and behaviors. We leave this for future work involving more controlled experiments, such as those using synthetic training and evaluation environments. In addition, we have added more experiments addressing the raised concerns in the rebuttal. ACG is presented as a hypothesis motivated by our observations. We also want to highlight the paper’s other core contributions, including the River recipe and the environment-auditing and filtering approach. The effectiveness of our pipeline is supported by strong benchmark results and in-depth experimental analyses.
W2. Contributions of the RIVER components
Thank you for this suggestion. We clarify the contributions through two complementary sets of ablations.
Behavior-verifier ablations
Section 3.1 already includes behavior-verifier ablations comparing outcome-reward-only GRPO with repetition-penalty and verification-reward variants on the same filtered environment set. The performance comparison is summarized below. Scores are averaged across the four benchmarks.
The repetition penalty improves average performance, whereas the verification reward provides little additional benefit. These results motivate including only the repetition penalty in the final RIVER recipe and show the effectiveness of the repetition verifiers (repetition penalty).
Environment-filtering ablations
Across the RL conditions in the following experiments and Appendix Table 8, we used identical behavior regularization. We evaluated RIVER-3.5K, Rubrics-3.5K, and Random-3.5K. For Rubrics-3.5K, we sampled 3.5K environments from the TMax environments that pass our rubric-based filtering. For Random-3.5K, we randomly sampled 3.5K environments from the TMax collection.
From these results, we observe that rubric-based filtering yields a substantial improvement in performance over the random baseline, showing the effectiveness of our rubric-based filtering system. The ablation on oracle filtering can be analyzed through the performance difference between RIVER-3.5K and Rubrics-3.5K. The core component of our filtering process is rubric-based filtering.
The motivation for our oracle-filtering process was to further remove some invalid tasks that never lead to non-zero rewards. For example, TMax task
task_001614_065db60dasks the agent to extract an ELF payload from a traffic log and recover its embedded C2 address, but the supplied log contains no payload bytes. The verifier nevertheless requires both artifacts, so the agent lacks the information needed to complete the intended procedure. Such cases motivate using strong oracle agents to flag potentially unsolvable tasks for exclusion, reducing useless training signals. This oracle-filtering process may also filter out hard problems and reduce the overall difficulty of the task sets.