acceptodds
Under review as a conference paper at ICLR 2027

TRACE: Capability-Targeted Agentic Training

Abstract

Models fail to complete agentic tasks often due to missing core capabilities. However, mainstream approaches for addressing these failures often rely on fine-tuning directly on the target environments or generating synthetic data that is not targeted to the model’s actual capability deficits, resulting in low sample efficiency and limited generalizability. We introduce TRACE (Turning Recurrent Agent failures into Capability targeted training Environments), an end-to-end system for environment specific agent self-improvement. TRACE contrasts successful and failed trajectories to automatically identify lacking capabilities, synthesizes a targeted training environment for each that rewards whether the capability was exercised, trains a LoRA adapter via RL on each synthetic environment, and then trains a mixture-of-experts (MoE) model over the capability adapters. TRACE can be effectively applied across different environments, improving over the base agent by +15.3 points on -Bench (customer service) and +15 points Pass@1 on SWE-bench Verified (software engineering), outperforming the strongest external baselines, GEPA and SWE-RL, by +8.6 points and +8.4 points, respectively. In addition, TRACE is more sample-efficient than strong finetuning baselines: using one-fourth the number of rollouts, TRACE outperforms the best-performing baselines, GRPO and GEPA, and achieves higher final accuracy by +10.4 and +8.6 points on -Bench.

Primary area:
reinforcement learning
Keywords:
Agentic Reinforcement Learning, Synthetic Environment Generation, LLM Agents, Self-improvement

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eEfNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TRACE, a framework that uses an analysis agent to compare successful and failed trajectories and identify recurring capability gaps. It then generates a dedicated training environment for each selected capability, trains separate LoRA adapters through reinforcement learning, and learns a router to combine them. Experiments show improvements over the evaluated baselines.

Critical strengths
  1. The paper addresses a useful problem of turning failed agent trajectories into targeted training environments for reinforcement learning.
  2. The method is clearly described, and the experiments are extensive. It clearly explains the full pipeline from capability discovery and environment generation to adapter training and routing, and is easy to follow.
Critical weaknesses
  1. Although the analysis is repeated multiple times, the novelty and robustness of using an analysis agent to discover failure patterns remain unclear. The identified capabilities may be task specific, and their organization is relatively flat. Human review of the discovered capabilities and experiments testing whether they transfer across tasks and domains would make the approach more convincing. A comparison with hierarchical capabilities, such as general capabilities and domain specific capabilities, would also be useful.

  2. It is unclear whether TRACE continues to improve when the number of capability adapters increases beyond four, and whether the router can still route effectively when more capabilities are introduced. Under a limited adapter budget, it would also be interesting to consider replacing adapters for capabilities that are no longer useful. A comparison between predefined capabilities and evolving capabilities could help evaluate this possibility.

  3. Some details needed to assess reproducibility and efficiency remain unclear, particularly the source and split of the router training data. The budget comparison should also account for environment generation, candidate filtering, and router training rather than relying mainly on rollout counts.

Questions for authors

See Weaknesses.

3 comments

Thank you for the suggestions on human validation, transfer, and how TRACE might work with a larger capability set.


W1: Capability diagnosis, human review, and transfer

TRACE turns recurring differences between successful and failed trajectories into executable training environments. We use a fixed capability dictionary across labeling runs and retain capabilities that consistently meet the coverage and contrastive-gap thresholds. For the ablation, we synthesize 4 capability environments per method on τ²-Bench (matching TRACE) and hold the rest of the pipeline fixed. Each method supplies capabilities at the same granularity as TRACE, a description plus example trajectories, so the comparison isolates the source of the capabilities.

τ²-Bench (Pass Rate)

ApproachAirline (%)Retail (%)Overall (%)
Base24.036.832.9
Generic environments36.040.439.0
Random-plausible36.041.239.6
Failure-only38.043.041.5
TRACE44.048.247.0
Human-written (manual ref.)46.050.949.4

Contrastive diagnosis improves the overall pass rate by 5.5 percentage points over failure-only diagnosis and comes within 2.4 points of the human-written reference. This auxiliary ablation uses task-level classifier routing. The main paper’s 48.2% overall pass rate uses the trained token-level MoE.

For the human-written reference, we spent about 5–8 hours reviewing all the rollouts used for diagnosis and identifying capability targets. The analysis agents completed this step substantially faster.

TRACE selected the capability targets automatically in the main experiments. We reviewed the generated environments for reuse of original scenarios and checked verifier decisions. Two authors independently audited 50 rollouts per environment and agreed with the verifier’s binary success judgment on all 200 audited rollouts. Measuring agreement between human and model capability labels would require a separate audit.

We evaluate agents adapted for τ²-Bench on ToolSandbox without retraining:

Training methodToolSandbox mean similarityPerfect scores
GRPO on τ²-Bench0.43019/129
TRACE on τ²-Bench0.48220/129

This comparison provides preliminary evidence of transfer to a different tool-use environment. We agree that comparing a flat capability set with a hierarchy of shared and domain-specific capabilities would help establish when that transfer occurs. The current experiments use a flat set.


W2: Adapter scaling and replacement

Our τ²-Bench scaling experiment covers up to four adapters. The submitted router selects one adapter per token, and its choice can change across an interaction. This keeps one capability adapter active per token, while a larger adapter bank would still increase storage and expand the choices the gate must learn to make.

The current experiments keep the chosen capability set fixed. The behavior of the router beyond four adapters and the value of replacing adapters as new deficits emerge remain open questions. We will scope the scaling claim to the range we evaluated.

Furthermore, given the limited task pools available for diagnosis in τ²-Bench and SWE-bench Verified, expanding the capability set through increasingly fine-grained subdivisions could produce capabilities supported by only a few tasks. We aimed to avoid this fragmentation and focus on reusable capabilities. This motivates our choice of capability granularity.


W3: Router data and efficiency

We train the router on all trajectories collected for adapter training in the synthetic environments, including successful and unsuccessful rollouts. This pool contains no original benchmark trajectories. We freeze the base model and adapters and optimize only the token-level gates with response-token cross-entropy and a load-balancing penalty. Reusing the synthetic pool requires no additional environment rollouts for router-data collection.

The 5,120 rollouts count adapter training across all capabilities. The approximately 2,500 episodes per environment in Appendix C belong to a separate run checking diversity and overlap. We will correct the wording in Appendix C and distinguish the adaptation rollout comparison from the additional costs of discovery, environment generation, candidate filtering, validation, and router optimization.

Should any important concern remain, we would be grateful if the reviewer could bring it to our attention during the discussion period. We would be happy to incorporate these clarifications and results into the paper and submit a revised PDF. If our response addresses the reviewer’s concerns, we would appreciate it if the reviewer would consider raising their score in light of the additional evidence.

Dear Reviewer eEfN,

Thank you again for your thoughtful feedback. As the rebuttal period draws to a close, we wanted to follow up on our response and highlight two new results relevant to your concerns about capability discovery:

  • The controlled ablation shows that contrastive diagnosis improves over failure-only diagnosis by 5.5 percentage points, coming within 2.4 points of the human-written reference but with less time spent.
  • The ToolSandbox evaluation provides preliminary evidence of transfer beyond τ²-Bench without retraining. We also clarified the router’s training data and the distinction between adaptation rollout counts and additional pipeline costs. We acknowledge that scaling beyond four adapters and evolving capability sets remain open questions, and will limit our claims accordingly.

If any concern remains unresolved, we would appreciate the opportunity to address it before the discussion ends. If these results and clarifications address your main concerns, we would be grateful if you could consider raising your score in light of the additional evidence.

Thank you for the detailed responses and additional experiments. But my main concerns remain: 1) TRACE only gets one more perfect score than GRPO on a relatively small task set, so the gain is quite modest 2) while I understand capping at 4 adapters as a trade-off, I am still concerned about how subjective the capability division is, and whether we need to redefine their number and scope for new benchmarks. Overall, I would like to maintain my original rating.

Reviewer_WXJAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TRACE, a framework for improving LLM agents through capability-targeted training. It identifies recurring capability deficits by contrasting successful and failed trajectories, synthesizes verifiable RL environments targeting these deficits, trains capability-specific LoRA adapters, and combines them using a learned MoE router. Experiments on τ²-Bench and SWE-bench Verified demonstrate substantial improvements over several adaptation baselines.

Critical strengths
  • Meaningful research direction. Identifying capability deficits from agent failures and constructing targeted training environments is a promising approach to agent self-improvement.

  • TRACE integrates capability discovery, environment synthesis, RL training, and adapter composition into a coherent pipeline.

  • The method demonstrates substantial improvements across different benchmarks and model backbones, supported by several useful ablation studies.

Critical weaknesses
  • Potential evaluation leakage. On τ²-Bench, capability discovery uses trajectories from tasks included in the final evaluation. Although the adapters are trained on newly synthesized environments, the training pipeline still benefits from failure information obtained from evaluation instances. This raises concerns about evaluation independence and whether the reported improvements generalize to genuinely unseen tasks. A strictly held-out evaluation would be important to validate the method's generalization claims.

  • Weak baseline performance relative to official reports. Qwen3.6-27B achieves 77.2% on SWE-bench Verified according to its official model card, whereas the paper reports only 68.0% for the base model and 73.2% after TRACE. Although evaluation harnesses differ, it remains unclear whether TRACE provides meaningful improvements under stronger, well-optimized agent configurations.

  • Insufficient isolation of the core contribution. The identified capabilities, such as structured data reasoning and tool-calling precision, are relatively general agent skills. It is unclear whether automatically discovering these deficits is necessary, or whether training on predefined general-purpose capabilities would achieve similar improvements. The gains may primarily come from targeted RL training and LoRA-MoE composition rather than capability discovery itself.

Questions for authors
  • Can the authors provide results on a strictly held-out τ²-Bench split, where evaluation instances are never used during capability discovery?
  • Why are the reported base-model scores substantially below official model-card results? Does TRACE still provide comparable gains under stronger agent harnesses and evaluation settings?
2 comments

Thank you for raising the evaluation overlap and for pointing us to the official Qwen result.


W1 & Q1: Held-out τ²-Bench evaluation

We evaluated the base agent, direct GRPO, and TRACE on the same 139 held-out tasks. Direct GRPO trains on the original tasks in the training split. TRACE uses that split for capability discovery and trains its adapters and router on trajectories from the synthesized environments.

BackboneBaseDirect GRPOTRACE
Qwen3-30B-A3B35.3% (49/139)31.7% (44/139)41.7% (58/139)
Qwen3.6-27B56.1% (78/139)51.8% (72/139)58.3% (81/139)

TRACE improves over the base agent by 6.5 and 2.2 percentage points, respectively, computed from the solve counts. Direct GRPO falls below the base agent for both backbones, a pattern consistent with overfitting to the small training split. The TRACE gains show that capability training can help on tasks excluded from discovery. We expect this transfer when the held-out tasks require capabilities identified during discovery. Failure modes absent from the training split can still go undiagnosed.

TRACE trains its adapters and router exclusively on synthetic rollouts and still outperforms GRPO-on-target and GEPA on the same 164 τ²-Bench tasks. GRPO-on-target trains the policy directly on those tasks, and GEPA uses their feedback to optimize prompts. The full evaluation includes tasks that informed TRACE’s capability discovery, so it measures benchmark-informed adaptation.


W2 & Q2: Comparison with the Qwen3.6-27B model card

Qwen reports 77.2% on SWE-bench Verified using its internal agent scaffold. We use mini-SWE-agent without modifying the scaffold, so the two scores come from different evaluation setups. Qwen3.6-27B model card

Within our mini-SWE-agent setup, TRACE improves Qwen3.6-27B from 68.0% to 73.2% (+5.2 percentage points). SWE-RL reaches 70.4% under the same setup. Measuring the gain under Qwen's internal scaffold would require a matched evaluation there.


W3: Contribution of automatic diagnosis

The capability names summarize more detailed descriptions and example trajectories that guide environment synthesis. For the ablation, we synthesize 4 capability environments per method on τ²-Bench (matching TRACE) and hold the rest of the pipeline fixed. Each method supplies capabilities at the same granularity as TRACE, a description plus example trajectories, so the comparison isolates the source of the capabilities.

τ²-Bench (Pass Rate)

ApproachAirline (%)Retail (%)Overall (%)
Base24.036.832.9
Generic environments36.040.439.0
Random-plausible36.041.239.6
Failure-only38.043.041.5
TRACE44.048.247.0
Human-written (manual ref.)46.050.949.4

Contrastive diagnosis improves the overall pass rate by 5.5 percentage points over failure-only diagnosis and comes within 2.4 points of the human-written reference. This auxiliary ablation uses task-level classifier routing. The main paper’s 48.2% overall pass rate uses the trained token-level MoE. The generic-environment result supplies a reference for broader training targets. The human-written reference shows what deliberate capability selection can achieve in the same ablation.

The submitted consolidation ablation addresses a different part of the pipeline. Given the same capability environments, training one adapter across them reaches 40.9%, while token-level composition reaches 48.2%. The two ablations provide evidence for both diagnosis and composition in their respective settings.

Should any important concern remain, we would be grateful if the reviewer could bring it to our attention during the discussion period. We would be happy to incorporate these clarifications and results into the paper and submit a revised PDF. If our response addresses the reviewer’s concerns, we would appreciate it if the reviewer would consider raising their score in light of the additional evidence.

Dear Reviewer WXJA,

Thank you again for your thoughtful feedback. As the rebuttal period draws to a close, we wanted to follow up on our response and highlight the additional evidence relevant to your main concerns:

  • Held-out evaluation: On 139 tasks excluded from capability discovery, TRACE improves over the base agent by 6.5 and 2.2 percentage points across the two backbones, while direct GRPO falls below the base agent on both.
  • Evaluation scaffold: Qwen’s official result uses its internal scaffold. Within our matched mini-SWE-agent setup, TRACE improves Qwen3.6-27B from 68.0% to 73.2%, compared with 70.4% for SWE-RL. Gains under Qwen’s internal scaffold remain untested.
  • Contribution of diagnosis: Holding the environment count and downstream pipeline fixed, TRACE outperforms generic-environment training by 8.0 percentage points and failure-only diagnosis by 5.5 points, supporting the contribution of contrastive capability discovery.

If any concern remains unresolved, we would appreciate the opportunity to address it before the discussion ends. If these results and clarifications address your main concerns, we would be grateful if you could consider raising your score in light of the additional evidence.

Reviewer_9kfmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TRACE, a framework that diagnoses recurring capability deficits by contrasting successful and failed agent trajectories, synthesizes capability-specific training environments, trains separate LoRA adapters with GRPO, and combines them through token-level routing. Experiments on customer-service and software-engineering tasks show improvements across two model backbones, including gains of 15.3 percentage points on τ²-Bench and 15 points on SWE-bench Verified for Qwen3-30B-A3B.

Critical strengths
  1. The framework addresses a concrete limitation of outcome-based agent training: heterogeneous failures provide limited information about which reusable behaviors require improvement. Connecting failure diagnosis to executable, verifiable training environments is a useful systems contribution.

  2. The empirical evaluation includes two backbones, multiple task families, and alternative capability-consolidation methods. The independent SWE-bench Verified evaluation is particularly valuable. The verifier audits and stochastic-decoding results provide additional supporting evidence.

Critical weaknesses
  1. The experiments establish the effectiveness of the complete pipeline more clearly than the contribution of contrastive capability diagnosis. Targeted task selection, reward shaping, multiple adapters, and routing change together. A controlled comparison against failure-only diagnosis or a fixed capability inventory is needed to substantiate the central mechanism. Additionally, τ²-Bench evaluation includes tasks used for capability discovery, limiting conclusions about generalization to unseen tasks (§4.1; Appendix B).

  2. The efficiency claim requires clearer accounting. It is unclear whether the reported rollout budget includes discovery, candidate-environment validation, all adapters, and router-data collection. The 5,120-rollout result also needs reconciliation with approximately 2,500 training episodes per environment in Appendix C. The source and selection of router-supervision trajectories are insufficiently specified for assessing data access and reproducibility.

Questions for authors
  1. How does performance differ between τ²-Bench tasks used for capability discovery and the remaining tasks? Under otherwise identical training conditions, how does contrastive diagnosis compare with failure-only diagnosis?

  2. What exactly is included in the rollout budget, how does it relate to the episode counts in Appendix C, and where do the router-training trajectories originate?

4 comments

Thank you for the careful review and for asking us to distinguish the effect of diagnosis from the rest of the pipeline.


W1 & Q1: Contrastive diagnosis versus failure-only diagnosis

Consider a failed trajectory that involves capabilities A, B, and C and a successful trajectory that involves A, D, and E. If the agent exercises A correctly in both, A is a weaker candidate for explaining the failure. A diagnosis based only on the failed trajectory could still select A as a training target. Comparing successes and failures helps us prioritize recurring deficits and avoid spending training rollouts on capabilities the agent already exercises successfully.

For the ablation, we synthesize 4 capability environments per method on τ²-Bench (matching TRACE) and hold the rest of the pipeline fixed. Each method supplies capabilities at the same granularity as TRACE, a description plus example trajectories, so the comparison isolates the source of the capabilities.

τ²-Bench (Pass Rate)

ApproachAirline (%)Retail (%)Overall (%)
Base24.036.832.9
Generic environments36.040.439.0
Random-plausible36.041.239.6
Failure-only38.043.041.5
TRACE44.048.247.0
Human-written (manual ref.)46.050.949.4

Contrastive diagnosis improves the overall pass rate by 5.5 percentage points over failure-only diagnosis and comes within 2.4 points of the human-written reference. This auxiliary ablation uses task-level classifier routing. The main paper’s 48.2% overall pass rate uses the trained token-level MoE.


W1 & Q1: Discovery tasks and held-out evaluation

On τ²-Bench, benchmark trajectories inform capability discovery. The identified deficits guide the creation of synthetic environments for adapter and router training. We manually checked these environments for copies of the original benchmark scenarios and found none in the environments we inspected.

TRACE trains its adapters and router exclusively on synthetic rollouts and still outperforms GRPO-on-target and GEPA on the same 164 τ²-Bench tasks. GRPO-on-target trains the policy directly on those tasks, and GEPA uses their feedback to optimize prompts. The full evaluation includes tasks that informed TRACE’s capability discovery, so it measures benchmark-informed adaptation. The aggregate scores do not provide a discovery-versus-remaining-task breakdown for that run. On SWE-bench, we use SWE-smith for discovery and training and the separate 500-instance Verified set for evaluation.

We also ran a separate split experiment. TRACE excluded 139 tasks from discovery, and direct GRPO trained on the original tasks in the training split. All three methods were scored on the same 139 held-out tasks:

BackboneBaseDirect GRPOTRACE
Qwen3-30B-A3B35.3% (49/139)31.7% (44/139)41.7% (58/139)
Qwen3.6-27B56.1% (78/139)51.8% (72/139)58.3% (81/139)

TRACE improves over the base by 6.5 and 2.2 percentage points, respectively, based on the solve counts.


W2 & Q2: Rollout counts and router training

  • 5,120 rollouts is the total for adapter training across all four capabilities in the reported τ²-Bench training curve.
  • Approximately 2,500 episodes per environment in Appendix C came from a separate run to examine scenario diversity and overlap. We will replace “training episodes” with “episodes collected in a separate validation run.”
  • Discovery and environment generation incur additional costs outside the 5,120-rollout count, and will specify in the revised version of the paper.

We train the router on all trajectories collected for adapter training in the synthetic environments, including successful and unsuccessful rollouts. This pool contains no original benchmark trajectories. We freeze the base model and adapters and optimize only the token-level gates with response-token cross-entropy and a load-balancing penalty. The gate requires no additional rollouts for data collection. The one-fourth comparison counts adaptation rollouts. Total compute would also include the stages above and gate optimization.

Should any important concern remain, we would be grateful if the reviewer could bring it to our attention during the discussion period. We would be happy to incorporate these clarifications and results into the paper and submit a revised PDF. If our response addresses the reviewer’s concerns, we would appreciate it if the reviewer would consider raising their score in light of the additional evidence.

Dear Reviewer 9kfm,

Thank you again for your thoughtful feedback. As the rebuttal period draws to a close, we wanted to follow up on our response and highlight the additional evidence relevant to your main concerns:

  • Contribution of diagnosis: With four environments per method and the downstream pipeline held fixed, TRACE outperforms failure-only diagnosis by 5.5 percentage points and generic-environment training by 8.0 points, supporting the contribution of contrastive capability discovery.
  • Held-out evaluation: On 139 tasks excluded from capability discovery, TRACE improves over the base agent by 6.5 and 2.2 percentage points across the two backbones, while direct GRPO falls below the base on both.
  • Rollout accounting and router data: The 5,120 rollouts cover training across all four adapters. The approximately 2,500 episodes in Appendix C belong to a separate validation process, not additional cost to the main pipeline. The router reuses synthetic adapter-training trajectories and requires no additional rollouts for data collection.

If any concern remains unresolved, we would appreciate the opportunity to address it before the discussion ends. If these results and clarifications address your main concerns, we would be grateful if you could consider raising your score in light of the additional evidence.

Thank you for the detailed responses and additional experiments. I will raise my scores accordingly.

Dear Reviewer 9kfm,

Thank you so much for your follow-up and for reconsidering your scores! We are glad that our additional experiments and clarifications addressed your concerns, and appreciate your recognition of TRACE's contribution in connecting failure diagnosis to verifiable training environments and the strength of our empirical evaluation across multiple backbones and task families. Thank you again for your thoughtful feedback and support for acceptance!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.