acceptodds
Under review as a conference paper at ICLR 2027

Learning Generalizable Behaviors for Terminal Agents

Abstract

Terminal agents are a compelling application of large language models (LLMs), with the potential to integrate deeply into users' daily workflows. Reinforcement learning (RL) is a key technique for improving their capabilities, making scalable training environments a central challenge. Since public real-user interaction data are scarce, synthetic environments provide a practical alternative, but often suffer from domain gaps and limited fidelity, leading to poor generalization. Existing work mainly scales the quantity and diversity of synthetic environments, while reward-signal quality and the mechanisms governing generalization remain under-explored. We study how RL improves terminal agents and propose the Agentic Compositional Generalization hypothesis: rather than teaching new domain-specific skills from scratch, RL primarily shapes high-level decision-making behaviors that compose and route low-level skills acquired during pre-training and supervised fine-tuning (SFT). This account is consistent with our empirical results and suggests that verifier quality, which determines which behaviors are reinforced, is more important than simply increasing environment quantity or diversity. Motivated by this insight, we propose River, a simple training recipe that improves reward quality by filtering low-quality environments and augmenting outcome rewards with process-level behavior regularization. Using this recipe, our RL-trained agent achieves the best performance among evaluated open-source RL-trained 8B models across four terminal-agent benchmarks. River also generalizes across model families, scales, agent harnesses, and RL objectives. Using fewer than 30% of the TMax training environments, River improves RL gains by 106% and 30% on average for models ranging from 2B to 27B on Terminal-Bench-Lite and Terminal-Bench-v2.1, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Coding Agents, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_opn3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies generalization in terminal agents trained with reinforcement learning (RL) in synthetic environments. It proposes the Agentic Compositional Generalization (ACG) hypothesis, which posits that pre-training and supervised fine-tuning (SFT) mainly provide low-level skills, whereas RL primarily improves how these skills are selected, composed, and coordinated across multi-turn interactions. The paper also introduces RIVER, which improves RL training signals through environment quality auditing, oracle-agent solvability filtering, and a repetition penalty. The approach is evaluated across multiple model scales, agent harnesses, and terminal-agent benchmarks. Skill-structure analysis, behavior–success prediction, SFT data ablations, and cross-domain RL experiments provide empirical support for the ACG hypothesis.

Critical strengths
  1. The training recipe is practically valuable and is supported by a relatively broad empirical evaluation. RIVER achieves positive results across multiple model scales, agent harnesses, and RL settings, providing useful evidence that improving environment quality and reducing low-value training data can benefit terminal-agent RL.

  2. The analysis offers an insightful and testable perspective. By combining skill-structure analysis, behavior prediction, SFT initialization studies, and cross-domain experiments, the paper provides a useful way to study what RL changes in terminal agents and demonstrates awareness of the distinction between correlational evidence and the effects of training interventions.

Critical weaknesses
  1. The mechanistic analysis is still largely based on indirect evidence. Chapter 4 provides relatively consistent support for the ACG hypothesis through skill-structure analysis, behavior features, and cross-domain experiments. However, the current definitions of “skills” and “behaviors” are primarily based on trajectory-level statistical features. These results are therefore best interpreted as empirical evidence consistent with the ACG hypothesis. A further discussion of the scope and limitations of these measurements would make the mechanistic interpretation more rigorous.

  2. The contributions of the individual RIVER components could be disentangled more clearly. The paper already includes a random-environment control and behavior-verifier ablations, making the overall experimental evaluation fairly comprehensive. However, RIVER combines environment filtering, pass@2 filtering, and a repetition penalty. More information would help clarify the relative contribution of each component, as well as how the difficulty and domain distribution of the environments change after filtering.

Questions for authors
  1. In Section 4.1, the skill labels primarily indicate whether a certain type of operation appears in a trajectory. Could the authors clarify whether the correctness of the resulting operation is also taken into account, and how the reliability of the GPT-5.4 skill annotations is validated?

  2. After rubric-based filtering and pass@2 filtering, did the difficulty and domain distribution of the environments change substantially? The appendix already provides some related statistics, but a further description of the overall task-distribution changes before and after filtering would help readers better understand whether the performance gains mainly come from improved reward integrity or from changes in the task set.

  3. Section 4.4 shows that RL training on a subset of domains can improve performance on other domains. Could the authors explain the rationale for selecting these domain subsets, and clarify whether the same trend holds for different domain combinations with the same number of training domains?

4 comments
AuthorsOpenReview ↗

Response [1/3]

We sincerely thank the reviewer for their time and effort in reviewing this paper. We would like to address their concerns and answer their questions in the following content.

W1. Scope and limitations of the mechanistic interpretation

A further discussion of the scope and limitations of these measurements would make the mechanistic interpretation more rigorous.

One limitation of our current analysis is that we cannot enumerate all skills and behaviors. We leave this for future work involving more controlled experiments, such as those using synthetic training and evaluation environments. In addition, we have added more experiments addressing the raised concerns in the rebuttal. ACG is presented as a hypothesis motivated by our observations. We also want to highlight the paper’s other core contributions, including the River recipe and the environment-auditing and filtering approach. The effectiveness of our pipeline is supported by strong benchmark results and in-depth experimental analyses.

W2. Contributions of the RIVER components

The contributions of the individual RIVER components could be disentangled more clearly. The paper already includes a ... More information would help clarify the relative contribution of each component, as well as how the difficulty and domain distribution of the environments change after filtering.

Thank you for this suggestion. We clarify the contributions through two complementary sets of ablations.

Behavior-verifier ablations

Section 3.1 already includes behavior-verifier ablations comparing outcome-reward-only GRPO with repetition-penalty and verification-reward variants on the same filtered environment set. The performance comparison is summarized below. Scores are averaged across the four benchmarks.

Training settingAvg. performance
GRPO with outcome reward only18.3
+ Repetition penalty19.4
+ Verification reward18.5

The repetition penalty improves average performance, whereas the verification reward provides little additional benefit. These results motivate including only the repetition penalty in the final RIVER recipe and show the effectiveness of the repetition verifiers (repetition penalty).

Environment-filtering ablations

Across the RL conditions in the following experiments and Appendix Table 8, we used identical behavior regularization. We evaluated RIVER-3.5K, Rubrics-3.5K, and Random-3.5K. For Rubrics-3.5K, we sampled 3.5K environments from the TMax environments that pass our rubric-based filtering. For Random-3.5K, we randomly sampled 3.5K environments from the TMax collection.

ModelOtblTbv2.1TbproTwvAvg.
RIVER-8B (ours)21.8 ± 2.99.7 ± 0.623.0 ± 3.123.0 ± 2.519.4 ± 1.2
RL on Rubrics-3.5K21.1 ± 1.89.0 ± 1.122.8 ± 1.523.0 ± 2.219.0 ± 0.7
RL on Random-3.5K18.6 ± 1.98.2 ± 1.323.7 ± 1.520.3 ± 0.617.7 ± 0.7
RIVER-8B-SFT11.3 ± 2.36.4 ± 1.313.7 ± 2.618.5 ± 2.312.5 ± 1.1

From these results, we observe that rubric-based filtering yields a substantial improvement in performance over the random baseline, showing the effectiveness of our rubric-based filtering system. The ablation on oracle filtering can be analyzed through the performance difference between RIVER-3.5K and Rubrics-3.5K. The core component of our filtering process is rubric-based filtering.

The motivation for our oracle-filtering process was to further remove some invalid tasks that never lead to non-zero rewards. For example, TMax task task_001614_065db60d asks the agent to extract an ELF payload from a traffic log and recover its embedded C2 address, but the supplied log contains no payload bytes. The verifier nevertheless requires both artifacts, so the agent lacks the information needed to complete the intended procedure. Such cases motivate using strong oracle agents to flag potentially unsolvable tasks for exclusion, reducing useless training signals. This oracle-filtering process may also filter out hard problems and reduce the overall difficulty of the task sets.

AuthorsOpenReview ↗

Response [2/3]

Q1. Skill correctness and annotation reliability

In Section 4.1, the skill labels primarily indicate whether a certain type of operation appears in a trajectory. Could the authors clarify whether the correctness of the resulting operation is also taken into account, and how the reliability of the GPT-5.4 skill annotations is validated?

Clarification of skill labels

We would like to clarify that, in the experiments supporting our ACG hypothesis in Section 4.1, skill-presence labels indicate that the corresponding skill is performed correctly. Also as defined on line 103, a skill is the ability to execute actions correctly.

Cross-model validation of the skill annotations

We used Claude Opus 4.6 to evaluate the GPT-5.4 skill annotations in 100 randomly sampled trajectories. The judgments were categorized as follows:

  • Supported: The annotation is judged correct and supported by direct evidence.
  • Unsupported: The annotation lacks direct supporting evidence.
  • Uncertain: The judge model cannot reach a clear determination.

The results are shown below. In this cross-model audit, 94.5% of the reviewed annotations were judged to be supported, providing additional model-based evidence for their reliability.

JudgmentPercentage
Supported94.5%
Unsupported4.1%
Uncertain1.4%

Q2. Changes in task difficulty and domain distribution

After rubric-based filtering and pass@2 filtering, did the difficulty and domain distribution of the environments change substantially? The appendix already provides some related statistics, but a further description of the overall task-distribution changes before and after filtering would help readers better understand whether the performance gains mainly come from improved reward integrity or from changes in the task set.

The task-category distributions of the original TMax collection and its filtered subsets are shown in the following table. Qualitatively, after pass@2 filtering, the difficulty decreases slightly because hard problems may also be filtered out.

Original TMax categoryBefore filteringAfter rubric filteringAfter rubric + pass@2
Data querying12.0%13.9%15.2%
Data science11.8%13.1%12.7%
Scientific computing11.7%11.0%8.9%
Software engineering11.5%7.2%6.5%
Debugging11.4%10.2%11.1%
File operations11.2%12.8%13.4%
Data processing10.8%15.6%17.0%
Security10.1%9.8%8.9%
System administration9.5%6.4%6.4%

The task-category distribution remains relatively stable after filtering.

In addition, in Section 4.4, Table 2, and Figure 7, we conducted controlled experiments by training on environments from different domains. We observed that, with environments containing reliable verifiers, the learned behaviors can generalize to different domains, including those excluded from training, and performance in those domains can also improve. These findings provide complementary evidence that the learned behaviors can transfer beyond the RL training domains.

AuthorsOpenReview ↗

Response [3/3]

Q3. Selection of training-domain subsets

Section 4.4 shows that RL training on a subset of domains can improve performance on other domains. Could the authors explain the rationale for selecting these domain subsets, and clarify whether the same trend holds for different domain combinations with the same number of training domains?

Domain partition

Our domain partition is based on tasks' core problem-solving requirements, with the aim of distinguishing domains by their substantive skill demands. Specifically, we identify skills using task instructions, skill metadata, and the code contained in successful reference trajectories, rather than relying on generic operations shared across domains, such as reading files or running Python. For example, tasks implemented in Python are distinguished according to whether their central computation involves graph algorithms, numerical computation, or data processing.

We consolidate the identified skills into eight domains and assign each task to the domain of its primary skill. This design emphasizes differences in the problems being solved and the core capabilities required, thereby promoting separation between domains in their primary skill requirements.

Evaluated training-domain subsets

RL (2 domains) is trained on Debug and Systems, whereas RL (4 domains) is trained on Data, Graph, Security, and Text. The two training-domain sets are entirely disjoint. As shown in the table below, both models outperform SFT across all eight evaluation domains, including those excluded from their respective RL training sets.

We selected the training-domain subsets based on a qualitative assessment of their primary task and skill requirements. Specifically, we chose Debug and Systems for the two-domain setting, and Data, Graph, Security, and Text for the four-domain setting, forming two disjoint training-domain subsets. The selection was guided by the aim of examining transfer across domains with relatively distinct core problem-solving requirements. Here, domain separation refers to differences in primary skill requirements; the domains may still share general operations and some underlying skills. The results are summarized in the following table.

ModelBuildDataDebugGraphNumericalSecuritySystemsTextOverall
RL on (Data, Graph, Security, Text)50.853.847.868.869.740.651.255.653.6
RL on (Debug, Systems)45.947.752.253.157.646.946.455.649.2
SFT34.429.243.528.145.521.926.244.432.3

The results show that the same trend of generalizable performance across different domains holds for different domain combinations.

Thanks for your response. I will keep my positive score.

Reviewer_JrhLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes the agentic compositional generalization hypothesis: RL mainly improves terminal agents by shaping behaviors that compose and route skills acquired before RL. Motivated by this view, the authors introduce RIVER, combining broader SFT skill coverage with rubric-based environment filtering, reference-agent solvability filtering, and repetition penalties applied to advantages. The resulting training pool contains approximately 3.5K environments. Experiments cover four terminal benchmarks and several Qwen model families and scales. The paper also studies skill-presence patterns, behavioral predictors of success, size-matched SFT datasets, and transfer to domains excluded from RL training.

Critical strengths
  1. The paper addresses a practical training problem with concrete evidence. The verifier-failure examples include both shortcuts that receive incorrect positive rewards and instruction-following solutions that are incorrectly rejected. These examples make the motivation for environment auditing quite clear.

  2. Several useful controls are already included. The SFT study compares datasets using same number of trajectories, and Table 8 includes a random 3.5K environment baseline. The domain-subset experiments also provide evidence that improvements can extend beyond the domains used during RL.

  3. The behavior study includes actual interventions rather than relying entirely on correlations. The comparison between repetition penalties and verification rewards is informative, especially since the verification intervention does not provide the same performance benefit.

  4. The evaluation covers four benchmarks and additional model families and scales, with seed variability reported for the main comparisons. The authors also disclose disagreement between audit models, which helps readers understand the limitations of the filtering process.

Critical weaknesses
  1. The mechanism interpretation is not fully identified by the current measurements. Sections 4.1 and E.1 measure binary skill presence, which does not capture correctness or proficiency. Preserved skill frequencies and co-occurrence therefore cannot rule out improvements in executing those skills. The behavior AUC comparison and repetition intervention are useful evidence, but they do not establish how much of the gain comes from routing versus improved skill execution. Also, the unsuccessful verification-reward experiment does not by itself show that verification has no causal benefit, especially given the read-only-inspection proxy in Table 9.

  2. The environment-filtering experiments do not fully separate reward quality from difficulty selection. Table 8 already provides a size-matched random baseline, but RIVER combines rubric filtering with a two-attempt reference-agent solvability filter. The latter changes the difficulty distribution by design. Rubric-only, oracle-only, and combined-filter comparisons with same training budget and regularization would help understand better which part produces the improvement.

  3. The reliability of the retained environment pool needs more direct validation. Appendix D.2 already acknowledges audit uncertainty, and only 6 of 15 initial CLEAN judgments are confirmed by the independent model. This disagreement is not a ground-truth error rate and does not directly describe the final 3.5K pool. However, passing the existing verifier with a reference agent also cannot exclude weak or shortcut-prone tests. A targeted audit of the final retained pool would strengthen the reward-integrity claim.

Questions for authors
  1. Could you evaluate atomic-skill correctness before and after RL, beyond whether the skill appears in a trajectory? This would help distinguish improved skill execution from changes in how skills are selected and composed.

  2. Could you provide rubric-only, oracle-only, and combined-filter ablations? Table 8 already controls environment count through the random 3.5K baseline, but the separate contributions of quality auditing and difficulty selection remain less clear. Please also clarify whether behavior regularization is identical across the Table 8 conditions.

  3. Have you performed human or execution-based adversarial checks on a random sample from the final retained 3.5K pool? Given the CLEAN-label disagreement in Table 7, this would give more confidence on the quality of the environments actually used for training.

  4. How closely does the verification reward follow the read-only-inspection definition in Table 9? Could its lack of performance gain reflect the proxy or shaping design, rather than verification itself having no causal benefit?

3 comments
AuthorsOpenReview ↗

Response [1/2]

We sincerely thank the reviewer for their time and effort in reviewing this paper. We would like to address their concerns and answer their questions in the following content.

Q1 / W1. Execution correctness and mechanistic interpretation

Could you evaluate atomic-skill correctness before and after RL, beyond whether the skill appears in a trajectory?

Atomic-skill correctness

We measured the skill correctness ratio on evaluation trajectories before and after RL. The task-averaged rate increased from ~70% before RL to ~80% after RL, as shown in the following table.

BenchmarkSFTRL
OTBL72.30%80.86%
TBv2.162.89%74.35%
TBpro70.35%83.96%
TWV65.46%78.26%

Predictive analysis with skill correctness

We also added the skill correctness ratio as an additional feature to Model-S in our ROC-AUC analysis. The results are shown below.

ModelSFT ROC-AUCRL ROC-AUC
Model-S (with skill correctness ratio)0.590.61
Model-B0.70.74

We can see Model-B's AUC still remains substantially higher than that of Model-S with the additional skill-correctness feature.

Q2 / W2. Ablations of the filtering components

Could you provide rubric-only, oracle-only, and combined-filter ablations? ... Please also clarify whether behavior regularization is identical across the Table 8 conditions.

Across the RL conditions in the following experiments and Appendix Table 8, we used identical behavior regularization. We evaluated RIVER-3.5K, Rubrics-3.5K, and Random-3.5K. For Rubrics-3.5K, we sampled 3.5K environments from the TMax environments that pass our rubric-based filtering. For Random-3.5K, we randomly sampled 3.5K environments from the TMax collection.

ModelOtblTbv2.1TbproTwvAvg.
RIVER-8B (ours)21.8 ± 2.99.7 ± 0.623.0 ± 3.123.0 ± 2.519.4 ± 1.2
RL on Rubrics-3.5K21.1 ± 1.89.0 ± 1.122.8 ± 1.523.0 ± 2.219.0 ± 0.7
RL on Random-3.5K18.6 ± 1.98.2 ± 1.323.7 ± 1.520.3 ± 0.617.7 ± 0.7
RIVER-8B-SFT11.3 ± 2.36.4 ± 1.313.7 ± 2.618.5 ± 2.312.5 ± 1.1

From these results, we observe that rubric-based filtering yields a substantial improvement in performance over the random baseline, showing the effectiveness of our rubric-based filtering system. The ablation on oracle filtering can be analyzed through the performance difference between RIVER-3.5K and Rubrics-3.5K. The core component of our filtering process is rubric-based filtering.

The motivation for our oracle-filtering process was to further remove some invalid tasks that never lead to non-zero rewards. For example, TMax task task_001614_065db60d asks the agent to extract an ELF payload from a traffic log and recover its embedded C2 address, but the supplied log contains no payload bytes. The verifier nevertheless requires both artifacts, so the agent lacks the information needed to complete the intended procedure. Such cases motivate using strong oracle agents to flag potentially unsolvable tasks for exclusion, reducing useless training signals. We acknowledge that oracle filtering may slightly decrease task difficulty, as hard but valid tasks may be excluded.

AuthorsOpenReview ↗

Response [2/2]

Q3 / W3. Adversarial checks on the retained environments

Have you performed human or execution-based adversarial checks on a random sample from the final retained 3.5K pool?

We performed adversarial attacks using Qwen3.6-27B on two sets of 50 tasks randomly sampled from the original TMax collection and the RIVER-filtered subset to compare verifier quality before and after filtering. The model was instructed to construct plausible solutions that deliberately omitted a required step or introduced a local semantic error while potentially passing the verifier. The rates at which these shortcuts passed the verifiers are reported below. The trajectory shortcuts were examined by another model, GPT-5.4. The results are shown in the following table.

CollectionVerifier-accepted shortcutsRate
TMax-Original918%
RIVER-Filtered24%

These results suggest that the RIVER-filtered subset is less susceptible to accepting shortcut solutions under the evaluation conditions.

Q4 / W1. Verification-reward implementation and interpretation

How closely does the verification reward follow the read-only-inspection definition in Table 9? Could its lack of performance gain reflect the proxy or shaping design, rather than verification itself having no causal benefit?

Verification-reward implementation

The verification reward follows the core idea of the read-only-inspection definition in Table 9, using a deterministic, command-based heuristic. Specifically, we examine commands issued within the three assistant turns immediately preceding the first <action>done</action> action and set verify_before_done = 1 if at least one command matches the inspection heuristic. We add a bonus of 0.05 to the trajectory-level advantage with this behavior verifier.

We agree that the lack of additional performance gains may reflect limitations of our verification proxy or reward-shaping design. As noted in our original discussion in Section 3.1, identifying behavior verifiers that improve performance is an interesting and non-trivial research direction. We will revise the corresponding statement to more explicitly clarify that this result does not rule out a causal benefit of verification itself and our conclusion concerns the specific implementation evaluated in our experiments.

Thanks for your reply, I will increase my score.

Reviewer_cXieOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the "agentic compositional generalization" hypothesis for terminal agents: pre-training and SFT provide low-level skills, while RL primarily changes how skills are selected, composed, and used across steps. To improve the behaviors learned during RL, the authors introduce RIVER, comprising SFT with broad skill coverage, a static rubric-based audit of task specifications/environments/verifiers, environment filtering using a strong model's pass@2, and an advantage penalty for steps with repetitive commands and similar feedback. The final training set contains approximately 3.5K filtered TMax environments.

The Qwen3-8B experiments achieve a mean score of 19.4 across four terminal benchmarks, compared with 17.8 for the strongest baseline in Table 1. The authors additionally report benefits from the filtered environments across 2B-27B models, another harness, and DPPO. The mechanism analyses examine skill presence and co-occurrence before and after RL, the predictive power of skill and behavior features, SFT datasets matched in size but differing in coverage, and transfer after restricting RL training domains.

Critical strengths
  1. There is broad evidence for the recipe's practical value. Table 1 reports a consistent mean advantage across four benchmarks, and Section 3.2 tests filtered environments at different model scales and training settings. Appendix D.4, Table 8, also includes a random subset of 3.5K environments, with a mean score of 17.7 versus 19.4 for full RIVER. The gains therefore cannot simply be attributed to reducing the environment count. The size-matched Diverse-DS versus Narrow-SFT comparison in Section 4.3 is also more convincing than comparing only 78K with 13.6K trajectories. These controls support the recipe's practical training value.
  2. The analysis of reward failures is concrete, and the effects of intervening on correlated behaviors are tested. Appendix D.4 shows cases where faulty verifiers reject correct solutions, as well as cases where leaked answers or weak tests reward incorrect solutions. Figure 3 further shows that the repetition penalty improves performance, whereas rewarding verification before completion changes that behavior's frequency without providing additional gains. This is a useful finding: a behavior that predicts success is not necessarily an effective target for a training intervention.
Critical weaknesses
  1. The mechanistic claim about compositional generalization exceeds what the current measurements establish. Section 4.1 uses a fixed taxonomy of 60 skills and annotates whether each skill appears in a complete trajectory. This binary presence variable measures neither action correctness nor the probability of executing a skill successfully under the same local state. Similar presence frequencies or skill co-occurrence structures therefore do not rule out improvements in the correctness, reliability, or conditional use of existing skills during RL. Section 4.2 compares binary skill presence with repetition, errors, verification before completion, and other features extracted from complete trajectories. The latter contain information arising alongside task execution outcomes, so their higher AUC does not establish the causal claim that RL mainly learns behaviors. The verification-reward experiment in Figure 3 demonstrates only that this reward implementation did not improve performance; it does not rule out a causal role for verification. This affects the paper's central contribution as a mechanism study. The claims should be framed as supporting evidence, or strengthened with direct interventions on skill correctness and behavioral transfer.
  2. Verifier quality, difficulty filtering, and the benefits of the complete recipe remain conflated. RIVER removes environments based on static audit labels and also removes those on which GPT-5.4 fails both attempts. The latter changes difficulty, infrastructure compatibility, and task distribution simultaneously. The random subset comparison in Appendix Table 8 is useful, but compares against full RIVER and does not isolate the contributions of the static audit, oracle filtering, and repetition penalty. Figure 4 uses TMax results from its original report, while Appendix B.2.2 acknowledges changes to the environment infrastructure, further limiting attribution to verifier quality. In the blind re-audit in Appendix D.2, only 6 of the 15 environments labeled CLEAN by GPT-5.4 remain CLEAN. This is not a human gold standard, but it indicates that the filtering labels are not yet stable. Comparisons within the same training stack that separate quality from difficulty are needed to fully support the conclusions about environment design.
Questions for authors
  1. Independent effect of quality filtering: With the same SFT initialization, training stack, optimization steps/sampling budget, and repetition penalty, can you compare a random 3.5K subset, static audit only, oracle filtering only, and their combination, while matching task domains and difficulty? Please clarify whether the random baseline in Appendix Table 8 also uses the repetition penalty. This would determine how much of the gains I attribute to verifier quality versus curriculum or behavior regularization.
  2. Mechanistic evidence: If skills are evaluated for execution correctness on the same local inputs, rather than merely for their presence, does skill preservation remain the dominant pattern before and after RL? Does behavior AUC remain substantially higher than that of skill-correctness features after controlling for task difficulty and trajectory length, and using only fixed execution prefixes? These measurements could distinguish improved skill reliability from changes in behavior across steps.
  3. Filtering reliability and resources: Have samples on which the two models disagree about CLEAN status undergone independent human or executable verification? Please report the residual verifier error rate in the final training set and the model-call/token costs of the static audit and oracle pass@2. This would affect the practical interpretation of "high-quality," "lightweight," and "efficient."
4 comments
AuthorsOpenReview ↗

Response [1/3]

We sincerely thank the reviewer for their time and effort in reviewing this paper. We would like to address their concerns and answer their questions in the following content.

W1. Mechanistic evidence for compositional generalization

The mechanistic claim about compositional generalization exceeds what the current measurements establish.

We have added more experiments addressing the raised concerns in the rebuttal. ACG is presented as a hypothesis motivated by our observations. We also want to highlight the paper’s other core contributions, including the River recipe and the environment-auditing and filtering approach. The effectiveness of our pipeline is supported by strong benchmark results and in-depth experimental analyses.

Clarification of skill labels

We would like to clarify that the skill labels indicate that a skill is performed correctly. As defined on line 103, a skill is the ability to execute actions correctly. We have revised the paper to emphasize more explicitly that skill-presence labels indicate that a skill is performed correctly.

Interpretation of the verification-reward experiment

The verification-reward experiment in Figure 3 demonstrates only that this reward implementation did not improve performance; it does not rule out a causal role for verification. This affects the paper's central contribution as a mechanism study.

Our intended conclusion concerns the our verification proxy and reward-shaping design evaluated in our experiments: it did not yield additional performance gains under the tested setup. As noted in our takeaway around line 297, discovering behavior verifiers that improve performance remains an interesting research direction. We will make this scope discussion more explicit in the revised paper.

The latter contain information arising alongside task execution outcomes, so their higher AUC does not establish the causal claim that RL mainly learns behaviors.

To remove the potential effects of information arising along side task execution outcomes, we perform AUC-ROC with Model-B by masking the task execution outcomes in the final turn. The results are presented in the following table. The results indicate that the predictive power of the behavior features does not come from the final-turn feedback. We interpret these AUCs as predictive associations consistent with our hypothesis, rather than as causal evidence.

SFTRL
Full trajectory0.700.74
Final execution feedback masked0.710.74

W2. Infrastructure differences in the TMax comparison

Figure 4 uses TMax results from its original report, while Appendix B.2.2 acknowledges changes to the environment infrastructure.

Regarding the infrastructure, the main difference is that we used locally hosted sandboxes instead of a remote sandbox service accessed via HTTP and APIs. We kept other training configurations, including RL algorithms, training code, harness, base models, and model sizes, unchanged to maintain a fair comparison.

AuthorsOpenReview ↗

Response [2/3]

Q1 / W2. Independent effect of quality filtering

With the same SFT initialization, training stack, optimization steps/sampling budget ... This would determine how much of the gains I attribute to verifier quality versus curriculum or behavior regularization.

In the following RL experiments and ramdom baseline in Appendix Table 8, behavior regularization is identical across the different RL environment sets. We also keep the same SFT initialization and training stack. We evaluated RIVER-3.5K, Rubrics-3.5K, and Random-3.5K. For Rubrics-3.5K, we randomly sampled 3.5K environments from the TMax environments that pass our rubric-based filtering. For Random-3.5K, we randomly sampled 3.5K environments from the TMax collection.

ModelOtblTbv2.1TbproTwvAvg.
RIVER-8B (full)21.8 ± 2.99.7 ± 0.623.0 ± 3.123.0 ± 2.519.4 ± 1.2
RL on Rubrics-3.5K21.1 ± 1.89.0 ± 1.122.8 ± 1.523.0 ± 2.219.0 ± 0.7
RL on Random-3.5K18.6 ± 1.98.2 ± 1.323.7 ± 1.520.3 ± 0.617.7 ± 0.7
RIVER-8B-SFT11.3 ± 2.36.4 ± 1.313.7 ± 2.618.5 ± 2.312.5 ± 1.1

From these results, we observe that rubric-based filtering yields a substantial improvement in performance over the random baseline, showing the effectiveness of our rubric-based filtering system. The ablation on oracle filtering can be analyzed through the performance difference between RIVER-3.5K and Rubrics-3.5K. The core component of our filtering process is rubric-based filtering.

The motivation for our oracle-filtering process was to further remove some invalid tasks that never lead to non-zero rewards. For example, TMax task task_001614_065db60d asks the agent to extract an ELF payload from a traffic log and recover its embedded C2 address, but the supplied log contains no payload bytes. The verifier nevertheless requires both artifacts, so the agent lacks the information needed to complete the intended procedure. Such cases motivate using strong oracle agents to flag potentially unsolvable tasks for exclusion, reducing useless training signals.

Task distribution.

The task category distribution of the original TMax collections and subsets after filtering are visualized in the following table. The task-category distribution remains relatively stable after filtering.

Original TMax categoryBefore filteringAfter rubric filteringAfter rubric + pass@2
Data querying12.0%13.9%15.2%
Data science11.8%13.1%12.7%
Scientific computing11.7%11.0%8.9%
Software engineering11.5%7.2%6.5%
Debugging11.4%10.2%11.1%
File operations11.2%12.8%13.4%
Data processing10.8%15.6%17.0%
Security10.1%9.8%8.9%
System administration9.5%6.4%6.4%

Q2 / W1. Mechanistic evidence

If skills are evaluated for execution correctness on the same local inputs, rather than merely for their presence, does skill preservation remain the dominant pattern before and after RL? Does behavior AUC remain substantially higher than that of skill-correctness features after controlling for task difficulty and trajectory length, and using only fixed execution prefixes? These measurements could distinguish improved skill reliability from changes in behavior across steps.

Adding skill correctness to Model-S

Using the similiar experiment settings in the original paper, we further added the skill correctness ratio as a feature to Model-S and evaluated its ROC-AUC on SFT model-generated and RL model-generated traces. The results, obtained using 15 seeds and 5-fold cross-validation, are shown below. We can see Model-B's AUC remain substantially higher than that of Model-S with the additional skill-correctness feature.

SFT ROC-AUCRL ROC-AUC
Model-S (with skill correctness ratio)0.590.61
Model-B0.700.74

Predictive analysis using capped trajectory windows

We additionally examine predictive associations with the observable context capped at KK saved assistant turns, for K∈{24,32,48}K \in \{24, 32, 48\} to make experiments under trajectory turn controlled. Features are computed only from the retained context, and the skill and behavior models use identical cross-validation folds within each condition. The results are shown in the following table.

KKTrajectory sourceModel-SModel-B
24SFT0.640.64
24RL0.540.71
32SFT0.600.69
32RL0.550.71
48SFT0.630.69
48RL0.580.73

Behavior features have higher predictive power than skill features in most evaluation results, which is consistent with our hypothesis. We interpret this as a supplementary predictive analysis showing that the tested behavior features have more predictive power than skill features after controlling for trajectory length and using only fixed execution prefixes.

AuthorsOpenReview ↗

Response [3/3]

As strictly controlling all evaluation conditions, such as task difficulty, is hard for real-world tasks, we leave further tests involving controllable task difficulty and local states in controlled synthetic training and evaluation tasks for future work. We acknowledge that this would help distinguish skill reliability from changes in skill selection and composition, and we will discuss this in the limitations and future work sections.

Q3 / W2. Filtering reliability and resource costs

Have samples on which the two models disagree about CLEAN status undergone independent human or executable verification? Please report the residual verifier error rate in the final training set and the model-call/token costs of the static audit and oracle pass@2.

Executable verification

We performed adversarial attacks using Qwen3.6-27B on two sets of 50 tasks randomly sampled from the original TMax collection and the RIVER-filtered subset to compare verifier quality before and after filtering. The model was instructed to construct plausible solutions that deliberately omitted a required step or introduced a local semantic error while potentially passing the verifier. The rates at which these shortcuts passed the verifiers are reported below. The trajectory shortcuts were examined by another model, GPT-5.4. The results are shown in the following table.

CollectionVerifier-accepted shortcutsRate
TMax-Original918%
RIVER-Filtered24%

These results suggest that the RIVER-filtered subset is less susceptible to accepting shortcut solutions under the evaluation conditions.

Filtering costs

The average token cost is USD 0.014 per input environment for rubric-based filtering (static audit) and USD 0.22 per input environment for oracle-based pass@2 filtering.

Area_Chair_GmvjOpenReview ↗

Reminder: Please Follow Up on Authors’ Responses

Dear Reviewer cXie,

A friendly reminder to check the authors’ responses and follow up with any comments or remaining concerns. Your feedback would be greatly appreciated.

Thank you for your time and effort!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.