acceptodds
Under review as a conference paper at ICLR 2027

TracePrism: Macro-to-Micro Diagnosis for Long-Horizon Targeted Agent Optimization

Abstract

Optimizing long-horizon LLM agents from execution traces requires deciding which failures to analyze, where they originate, and which components to revise. Large trace collections contain recurring failures, while long executions can obscure the steps and component interactions underlying an observed failure. We present TracePrism, a macro-to-micro framework for budget-aware bottleneck localization and agent optimization. At the macro level, TracePrism groups failed trajectories by execution structure and failure information, then selects representative cases under a fixed trajectory budget for detailed diagnosis. At the micro level, it follows data and control dependencies backward from failure manifestations to construct diagnostic slices, localize plausible root-cause steps, and identify editable intervention targets. These diagnoses guide component-specific revisions that are checked against component responsibilities and applied jointly, leaving untargeted specifications unchanged. Across VeruSAGE-Bench, WebArena, and HotpotQA, TracePrism achieves the highest mean task performance among the evaluated methods. On VeruSAGE-Bench, it improves the success rate of an expert-designed agent from 46.2% to 58.8%, a gain of 12.6 percentage points. Scaling experiments on this benchmark further show a favorable trade-off between task performance and optimizer-side LLM cost as the failure corpus grows. The code is available at https://anonymous.4open.science/r/TracePrism-8CC3.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, Agent Optimization, Long-horizon Agents, Trajectory analysis, Context Engineering, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_Ffm9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TracePrism improves agent component instructions using a four-stage diagnosis pipeline. It profiles and clusters failed trajectories, selects representative cases under a deep-diagnosis budget, constructs backward slices following inferred data/control dependencies, maps plausible failure causes to editable components, and applies scoped instruction revisions. On VeruSAGE-Bench, WebArena, and HotpotQA, it obtains the highest mean performance among the evaluated methods, including a VeruSAGE success increase from 46.2% to 58.8% across three independent optimization runs. Additional experiments examine optimizer-side cost as the failure corpus grows, selection and diagnostic-context ablations, and agent/step attribution on Who&When.

Critical strengths
  1. Meaningful end-to-end improvements across diverse tasks.

TracePrism demonstrates consistent improvements across formal verification, browser interaction, and multi-hop QA, with three independent optimization runs in the main evaluation. On VeruSAGE, it improves performance by 12.6 percentage points over the base agent and 11.0 points over the strongest evaluated baseline, with additional gains on WebArena and HotpotQA. These results demonstrate the effectiveness of the framework across different task structures rather than a single specialized workflow.

  1. A well-motivated framework for identifying actionable optimization opportunities.

The paper presents a coherent approach to analyzing agent failures by separating failure selection, upstream evidence tracing, and identification of the components responsible for errors. This addresses an important challenge in modular agents: the component where a failure becomes visible is not necessarily the one responsible for the underlying decision. The framework provides a practical way to identify where optimization efforts should be focused.

Critical weaknesses
  1. Limited evaluation on modern long-horizon agents.

The main experiments use o4-mini and GPT-4o backbones, which may not fully reflect the capabilities of stronger, more recent agents. Additionally, HotpotQA uses a relatively simple four-stage workflow, while WebArena has only one editable instruction, making it difficult to evaluate the framework’s ability to identify the right component to optimize. Experiments with stronger backbones and more complex, long-horizon coding or terminal tasks would help demonstrate the framework’s generalizability to modern agentic workflows.

  1. Limited evidence supporting the effectiveness of individual components.

The diagnostic ablations are relatively small, using only ten failed trajectories and a 50-task evaluation subset with a single run per setting. Notably, both Current-context and Full-trace Repair achieve 54% success, compared to 56% for the full pipeline—a difference of just one task. These results make it difficult to determine whether the proposed diagnosis mechanism provides a meaningful advantage over simpler alternatives. Larger-scale ablations with multiple runs would help establish the contribution of each component.

Questions for authors
  1. Budget comparability: How would the main results change if all methods were given the same eligible traces, editable components, and total optimization budget, including profiling, clustering, diagnosis, and policy revision?
  2. Effectiveness of the diagnosis mechanism: Does the 54% vs. 56% performance difference hold on larger test sets or across multiple runs? Could the authors separately ablate evidence slicing and target attribution while keeping trace selection and revision procedures fixed?
  3. Generalization to long-horizon agents: Have the authors evaluated TracePrism on more complex coding or terminal benchmarks, such as Terminal-Bench or SWE-bench, using stronger, more recent model backbones? Such results would help establish whether the improvements generalize to modern long-horizon agent workflows.
5 comments
AuthorsOpenReview ↗

Response to Reviewer Ffm9 (1/2)

We sincerely thank Reviewer Ffm9 for the insightful feedback and constructive suggestions. Below, we address W1–W2 and Q1–Q3 with new experimental results on SWE-bench Pro using modern model backbones, alongside our new 3-seed independent ablations on VeruSAGE-Bench.

W1 & Q3: Generalization to modern long-horizon benchmarks and agents

To evaluate whether TracePrism generalizes to complex, long-horizon software engineering agents powered by stronger, more recent backbones, we conducted two complementary sets of experiments on SWE-bench Pro and compared TracePrism against GEPA, a prominent agent optimization baseline. Both evaluations use SWE-agent as the base agent, GPT-5.6-Luna as the target model, and GPT-6-Astra as the optimizer across distinct data splits and reasoning regimes.

1. Stratified Full-Benchmark Evaluation (GPT-5.6-Luna, max reasoning effort)

First, we partitioned the full SWE-bench Pro benchmark using a repository-stratified split into 292 training tasks, 145 validation tasks, and 294 held-out test tasks across 11 repositories:

RepositoryTasksBase AgentGEPATracePrism
NodeBB/NodeBB1788.24%76.47%88.24%
ansible/ansible3997.44%87.18%94.87%
element-hq/element-web2382.61%73.91%86.96%
flipt-io/flipt3450.00%52.94%58.82%
future-architect/vuls2580.00%84.00%84.00%
gravitational/teleport3187.10%90.32%93.55%
internetarchive/openlibrary3786.49%91.89%94.59%
navidrome/navidrome2378.26%86.96%78.26%
protonmail/webclients2684.62%84.62%92.31%
qutebrowser/qutebrowser3180.65%100.00%87.10%
tutao/tutanota862.50%50.00%50.00%
Overall29480.95%82.31%85.03%

Even the base agent already solves 80.95% of tasks, TracePrism improves the overall success rate by +4.08 percentage points over the base agent (+12 solved tasks) and +2.72 percentage points over GEPA (+8 solved tasks), achieving the highest or tied-highest score on 7 of the 11 repositories.

2. Cross-Repository Transfer Evaluation (GPT-5.6-Luna, low reasoning effort):

Second, to test cross-repository generalization under a tighter reasoning budget, we optimized policies solely on failed trajectories from qutebrowser and evaluated them on 237 held-out test tasks across three unseen repositories:

Methodansible/ansible (96)flipt-io/flipt (85)element-hq/element-web (56)Overall (237 Tasks)
Base Agent45.83%16.47%37.50%33.33%
GEPA44.79%18.82%37.50%33.76%
TracePrism47.92%22.35%42.86%37.55%

TracePrism improves the overall success rate by +4.22 percentage points over the base agent (+10 solved tasks) and +3.80 percentage points over GEPA (+9 solved tasks), outperforming both baselines across every evaluated repository.

W2 & Q2: Larger-scale 3-seed independent ablations

To verify whether the diagnostic gains hold on larger test sets and across multiple runs, and to separately ablate evidence slicing (Phase 2) and target attribution (Phase 3), we conducted new 3-seed ablations on VeruSAGE-Bench (50-task training pool, 200 test tasks; target agent: gpt-o4-mini). Due to API quota limits, we unified the optimizer model across all settings in these new experiments to GPT-6.1-sol with high thinking effort, keeping the optimizer configuration consistent.

Ablation SettingMean SR ± stdMean Cost
TracePrism (Full Pipeline)52.83 ± 1.53%$4.68
w/o Selection (All Traces)45.83 ± 1.04%$13.26
Random Selection46.83 ± 0.76%$5.29
Full Trace + Attribution (w/o Phase 2)45.33 ± 1.53%$12.97
Full Trace + Manifestation (w/o Phases 2 & 3)46.17 ± 1.53%$12.14
Sliced Trace + Manifestation (w/o Phase 3)43.50 ± 1.00%$4.44
AuthorsOpenReview ↗

Response to Reviewer Ffm9 (2/2)

These results establish two clear conclusions:

  1. The diagnostic advantage is supported by a larger, repeated evaluation: In our new three-run evaluation on 200 test tasks, TracePrism achieves a mean success rate of 52.83 ± 1.53%, outperforming Full Trace + Manifestation by 6.67 percentage points, while reducing optimizer cost by 61.4%. These results strengthen the empirical support for the effectiveness and optimizer-side efficiency of the proposed diagnostic mechanisms.
  2. Independent contributions of Evidence Slicing and Target Attribution: Removing either stage while keeping trace selection and revision procedures fixed severely degrades performance. Without evidence slicing (Full Trace + Attribution, -7.50% SR), long-horizon irrelevant steps and retry noise overwhelm raw-trace attribution and inflate token overhead; without target attribution (Sliced Trace + Manifestation, -9.33% SR), revising the error-manifestation site rather than the upstream responsible component leaves underlying decision flaws unresolved.

Q1: Budget Comparability & End-to-End Cost Accounting

To address budget comparability, we conducted an additional controlled comparison between TracePrism and GEPA on HotpotQA, using 200 training examples, 50 validation examples, and 300 held-out test examples. Both methods were given the same pool of eligible traces, identical editable components, and the same $2.00 total optimization budget cap, covering all optimization-stage costs, including trace collection, profiling, diagnosis, policy revision, and candidate evaluation.

We chose HotpotQA for this experiment because multi-round optimization requires repeated agent executions for trace collection and candidate evaluation, making budget-matched experiments on VeruSAGE substantially more time-consuming. Even on HotpotQA, GEPA required 6 hours and 9 minutes to complete its run.

MethodOptimization RoundsAgent CallsBest Validation ScoreTest Score
GEPA201,24056.0%61.0% (183/300)
TracePrism125058.0%64.0% (192/300)

GEPA's final candidate was selected based on the highest validation score from its Pareto frontier. Under the same $2.00 optimization budget, TracePrism outperforms GEPA by 3.0 percentage points on the held-out test set, solving 9 additional tasks while requiring 79.8% fewer agent calls. These results demonstrate that TracePrism maintains its performance advantage when eligible traces, editable components, and total optimization budget are matched.

We have updated the revised manuscript with all clarifications and new experimental results above, highlighted in blue for ease of review. Thank you again for your time and constructive feedback.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer,

We hope this message finds you well. As the discussion period draws to a close, we wanted to follow up on our response to your review. If any of your concerns remain or if anything in our response is unclear, we’d be happy to clarify. Your insights are invaluable to us.

Thank you for taking the time to review our paper.

Best regards,

The Authors

Reviewer_H7X9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

TracePrism uses failed execution traces to improve the policies of agent components. It groups failures and selects a small set of examples, traces backward from an observed failure to relevant earlier steps, identifies which components to revise, and updates their prompts or policies. The default budget allows up to five clusters and five traces per cluster, with 25 traces as the upper limit for detailed analysis. The authors evaluate the optimized systems on VeruSAGE-Bench, WebArena, and HotpotQA, and assess failure localization on Who&When. The overall gains are encouraging, but the added value of the proposed selection and diagnosis procedure over simpler ways of generating component-policy revisions remains unclear.

Critical strengths
  1. The complete workflow delivers substantial empirical gains. On VeruSAGE-Bench, success increases from 46.2% to 58.8%, exceeding the strongest evaluated baseline at 47.8%, with improvements across all five projects. On WebArena, TracePrism reaches 22.0%, compared with 15.1% for the strongest baseline. Table 2 also suggests a potential optimizer-side cost advantage over Full-trace Repair ($2.96 versus $5.93), although this single-run result needs replication. These results are the strongest evidence for the practical usefulness of the complete method.

  2. The method connects failure diagnosis to actionable component revisions. It distinguishes the component where an error appears from the component responsible for the problematic behavior, and checks that proposed revisions fit the target's responsibilities and available context. The human-annotated Who&When evaluation provides separate evidence for agent and failure-step localization. This combination supports the value of the diagnostic approach and its relevance to improving modular agents.

Critical weaknesses
  1. The added value of explicit diagnosis is not sufficiently isolated. Current-context and Full-trace Repair only revise components where failures manifest; neither provides a matched comparison with direct revision of an upstream controller from the same traces. These repair ablations also remove slicing and target attribution together, with success increasing from 54% to 56% in one run. The perturbation study measures context sensitivity without establishing the advantage of dependency-guided slicing. Consequently, the experiments leave unclear whether the proposed diagnostic stages improve policy revision beyond a simpler direct-revision procedure, limiting support for a central methodological contribution.

  2. Evidence for representative selection remains limited. Table 2 includes a random-selection control, but uses a pool of only 10 failed traces and one run per setting: representative selection reaches 56% success versus 52% for random selection. The scaling experiment extends to 453 traces but lacks an equal-size random-selection control. Thus, the evidence does not establish whether the proposed selection strategy provides a reliable advantage as the failure corpus grows, which is central to the method's budget-aware optimization claim.

Questions for authors
  1. Compare direct controller revision with revision guided by explicit slicing and diagnosis, using the same failed traces, optimizer, and budget. Fix the controller as the only editable component and generate revisions independently for both conditions.

  2. Compare representative selection with equal-size random sampling from the full 453-trace corpus. Repeat sampling and optimization, and report selected counts, mean performance, variability, and costs.

  3. Evaluate slicing and target attribution independently by crossing full versus sliced traces with manifestation-based versus attributed edit targets. Keep the remaining settings fixed, repeat the comparisons, and report performance and optimization costs, including candidate executions.

2 comments
AuthorsOpenReview ↗

Response to Reviewer H7X9

We sincerely thank Reviewer H7X9 for the thorough assessment and constructive experimental suggestions. Following your questions (Q1–Q3), we conducted new controlled experiments on VeruSAGE-Bench with an expanded 200-task test set (target agent: gpt-o4-mini). Due to API quota limits, we unified the optimizer model across all settings in these new experiments to GPT-6.1-sol with high thinking effort, ensuring a strictly fair comparison.

Q1 & W1: Direct Controller Revision vs. TracePrism

Following Q1 and W1, we fixed the upstream controller as the sole editable component and compared direct controller revision against controller revision guided by explicit slicing and diagnosis using the exact same failed traces across 3 seeds (50-task training pool, 200 test tasks). We also include the full TracePrism pipeline, which additionally allows dynamic target attribution across all editable components for reference:

Revision ConditionEditable ScopeMean SR ± std
Controller only (Direct revision)Controller only45.33 ± 2.02%
Controller only (Slicing + diagnosis)Controller only46.50 ± 1.32%
TracePrism (Slicing + diagnosis + dynamic attribution)All components52.83 ± 1.53%

As shown in the table above, when edits are restricted to the controller, explicit slicing and diagnosis improve the mean success rate from 45.33 ± 2.02% to 46.50 ± 1.32% (+1.17 percentage points), suggesting that dependency-guided slicing helps reduce noise from downstream retries. However, controller-only revisions cannot address failures originating from subagent-level execution flaws. Allowing TracePrism to dynamically attribute failures and apply scoped instruction revisions to the responsible components yields an additional +6.33 percentage points. These results show that TracePrism's gains do not simply come from revising the controller, but also benefit from identifying and updating the appropriate components.

Q2 & W2: Representative Selection vs. Random Sampling on the Full Training Corpus

Following W2 and Q2, we compared representative selection against equal-size random sampling drawn from the full disjoint training corpus to examine whether our selection strategy provides a reliable advantage as the training corpus scales. Because we expanded the held-out test set from 50 to 200 tasks for stronger statistical reliability, the remaining training corpus contains 303 traces instead of 453. Both methods selected 22 traces per run from the 303-trace training corpus. We report the mean success rate and sample standard deviation across three independent optimization runs, evaluated on the same 200 held-out tasks.

Selection Strategy (303 traces)Mean SR ± stdMean Cost
Equal-size random sampling47.67 ± 0.76%$6.57
Representative selection (TracePrism)53.83 ± 2.02%$6.23

On the full 303-trace corpus, representative selection outperforms random sampling by +6.17 percentage points on average, while reducing optimization cost by $0.34. These results demonstrate that the advantage of representative selection persists as the training corpus grows, achieving higher success rates than random sampling under the same trajectory budget.

Q3 & W1: Independent Evaluation of Slicing and Target Attribution

Following Q3 and W1, we evaluated diagnostic trace slicing (phase 2) and intervention target attribution (phase 3) independently by crossing full versus sliced traces with manifestation-based versus attributed edit targets across 3 seeds (50-task training pool, 200 test tasks), keeping all remaining settings fixed:

Trace ContextEdit TargetSettingsMean SR ± stdMean Cost
Full TraceManifestationw/o Phases 2 & 346.17 ± 1.53%$12.14
Full TraceAttributedw/o Phase 245.33 ± 1.53%$12.97
Sliced TraceManifestationw/o Phase 343.50 ± 1.00%$4.44
Sliced TraceAttributedTracePrism52.83 ± 1.53%$4.68

These controlled ablations demonstrate the complementary roles of diagnostic slicing and target attribution:

  • Attribution without Slicing (Full Trace + Attributed: 45.33%, $12.97): Costs approximately 2.8× as much while reducing SR by -7.50% compared to TracePrism, suggesting that irrelevant long-horizon context interferes with accurate attribution.
  • Slicing without Target Attribution (Sliced Trace + Manifestation: 43.50%, $4.44): Reduces cost, but revising the manifestation component rather than the upstream root-cause target degrades SR by -9.33%, supporting that both stages are jointly required.

We have updated the revised manuscript with all clarifications and new experimental results above, highlighted in blue for ease of review. Thank you again for your time and constructive feedback.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer,

We hope this message finds you well. As the discussion period draws to a close, we wanted to follow up on our response to your review. If any of your concerns remain or if anything in our response is unclear, we’d be happy to clarify. Your insights are invaluable to us.

Thank you for taking the time to review our paper.

Best regards,

The Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8vwq (1/2)

We sincerely thank Reviewer 8vwq for the thorough review and valuable pointers. Below, we address W1 and W2 with clarified distinctions from prior work and new 3-seed controlled experiments.

W1: Limited methodological novelty.

Thanks for your feedback, we clarify below what distinguishes TracePrism from existing methods and why its design is necessary.

TracePrism resolves a fundamental structural problem in long-horizon, modular agents that existing methods overlook: the misalignment between where a failure manifests and which component is architecturally eligible and contextually equipped to fix it.

Existing agent optimization methods, such as TextGrad and GEPA, operate over full or summarized trajectories and do not isolate dependency slices or check component observability. Meanwhile, existing diagnostic work, such as Who&When and AgentRx, focuses on post-hoc failure attribution without mapping diagnoses to actionable, editable intervention targets. TracePrism introduces three tightly coupled mechanisms to bridge this gap:

  • Two-Stage Attribution with Component Observability Checks (Phases 3–4): We explicitly decouple plausible root-cause step localization from editable intervention target mapping. Crucially, Phase 4 enforces Component Responsibility & Observability Checks—verifying whether a proposed rule is executable given a component's state visibility (e.g., prohibiting stateless executors from receiving cross-turn history rules and reassigning them to the stateful router).
  • Position-Preserving Dependency-Guided Backward Slicing (Phase 2): Instead of lossy summarization, TracePrism traces data and control dependencies backward from the task-relevant failure manifestation, retaining normally completed upstream steps whose outputs or routing decisions induced downstream failures while preserving exact original trace indices for uncorrupted inspection.
  • Structure-Constrained Macro Profiling & Budget Decoupling (Phase 1): Using LLM-synthesized deterministic Python profilers, TracePrism groups traces by strict execution-structure signatures and local error rates, decoupling corpus coverage from fixed diagnostic budget.

The new ablation results below further demonstrate the contributions and complementary roles of these mechanisms in agent optimization.

W2: Insufficient experimental evidence supporting the proposed methodology.

We thank the reviewer for these constructive suggestions. Below we provide new controlled experiments and clarifications for all three empirical concerns: (1) isolating Phase 2 and Phase 3 over multiple seeds on a larger test set, (2) budget-matched comparisons, and (3) validating target attribution on VeruSAGE-Bench:

1. Comprehensive Independent Ablations on a 200-Task Test Set:

To separate the individual contributions of Phase 2 (Diagnostic Trace Slicing) and Phase 3 (Intervention Target Attribution) and overcome the single-run and small test set limitations of the original ablation study, we conducted a new 3-seed ablation on VeruSAGE-Bench with larger training and test sets (50 training / 200 test tasks, target agent: gpt-o4-mini). Due to the API quota limits, we changed the optimizer model across all ablation settings to GPT-6.1-sol with high thinking effort, ensuring a strictly controlled and fair comparison.

Ablation SettingMean SR ± stdMean Cost
TracePrism (Full Pipeline)52.83 ± 1.53%$4.68
w/o Selection (All Traces)45.83 ± 1.04%$13.26
Random Selection46.83 ± 0.76%$5.29
Full Trace + Attribution (w/o Phase 2)45.33 ± 1.53%$12.97
Full Trace + Manifestation (w/o Phases 2 & 3)46.17 ± 1.53%$12.14
Sliced Trace + Manifestation (w/o Phase 3)43.50 ± 1.00%$4.44

These independent ablations demonstrate the complementary roles of diagnostic slicing and target attribution:

  • Attribution without Slicing (Full Trace + Attribution): Costs nearly 3×3\times more ($12.97) yet drops SR by -7.50% (45.33%), showing that long-horizon irrelevant context overwhelms raw-trace attribution.
  • Slicing without Target Attribution (Sliced Trace + Manifestation): Reduces a little cost ($4.44) but drops SR by -9.33% (43.50%), confirming that updating the manifestation component rather than the upstream root-cause target harms execution.
  • Unselective Optimization (w/o Selection): Costs $13.26 while dropping SR by -7.00% (45.83%) due to redundant failure noise.
AuthorsOpenReview ↗

Response to Reviewer 8vwq (2/2)

2. Budget-Matched Comparisons:

To address the concern about unmatched computational budgets, we conducted an additional budget-matched comparison between TracePrism and GEPA on HotpotQA, using 200 training examples, 50 validation examples, and 300 held-out test examples. Both methods were given the same pool of eligible traces, identical editable components, and the same $2.00 total optimization budget, covering all optimization-stage costs, including trace collection, profiling, diagnosis, policy revision, and candidate evaluation.

We chose HotpotQA because multi-round optimization requires repeated agent executions and trace collection on the training and validation sets, making equivalent experiments on VeruSAGE-Bench substantially more time-consuming. Even on HotpotQA, GEPA required 6 hours and 9 minutes to complete 20 optimization rounds. This additional comparison complements our VeruSAGE-Bench experiments by directly evaluating performance under matched computational budgets.

MethodOptimization RoundsAgent CallsBest Validation ScoreTest Score
GEPA201,24056.0%61.0% (183/300)
TracePrism125058.0%64.0% (192/300)

Under the same optimization budget, TracePrism outperforms GEPA by 3.0 percentage points on the held-out test set, solving 9 additional tasks while using 79.8% fewer agent calls. These results demonstrate that TracePrism maintains its performance advantage under a matched monetary budget, rather than benefiting from a larger optimization expenditure.

3. Validation of Intervention Targets on VeruSAGE-Bench:

We agree that the Who&When evaluation does not directly validate intervention-target correctness on VeruSAGE-Bench. VeruSAGE-Bench does not provide intervention-target annotations, and we have not completed a dedicated expert-annotation study within the rebuttal period. We therefore provide two complementary analyses of intervention utility.

First, our new controlled ablation evaluates the contribution of target attribution. With the same sliced traces, revising attributed targets achieves 52.83 ± 1.53% success, compared with 43.50 ± 1.00% when revising manifestation-based targets, across three runs on 200 held-out tasks. This 9.33-percentage-point improvement supports the practical value of diagnosis-guided target selection over simply revising the components where failures manifest.

Second, a task-level transition analysis from our main evaluation on VeruSAGE-Bench provides supplementary evidence of repair benefits and side effects. On the same 106 held-out tasks of main experiments, we compared each optimized policy against the Base Agent:

MethodWrong → CorrectCorrect → WrongNet Gain
GEPA94+5
TracePrism170+17

TracePrism repairs 17 of the 61 initially failed tasks while preserving all 45 initially successful tasks. Its improvement therefore comes entirely from newly solved tasks, with no observed regressions in this evaluation.

Together, the controlled ablation supports the utility of intervention-target selection, while the task-level analysis documents the benefits and observed side effects of the resulting scoped instruction revisions. These results provide complementary evidence for our diagnostic-to-intervention design.

We have updated the revised manuscript with all clarifications and new experimental results above, highlighted in blue for ease of review. Thank you again for your time and constructive feedback.

AuthorsOpenReview ↗

Summary of the Discussion

Dear Area Chair and Reviewers,

We thank the Area Chair and all reviewers for their time and constructive comments, which have made this paper more complete. During the discussion, Reviewer Ffm9 confirmed that the additional experiments and clarifications resolved all of their concerns and raised their rating from 2 to 3 (Weak acceptance). The additions are highlighted in blue in the revised manuscript.

The reviewers asked for larger and more rigorous ablation studies, clearer evidence of each component's contribution, and an evaluation of whether TracePrism generalizes to modern coding agents. We addressed these questions with new controlled experiments and comparisons, as summarized below:

  • Component contributions on a larger test set. In a three-seed ablation on 200 held-out VeruSAGE-Bench tasks, the full method achieved a mean success rate of 52.83 ± 1.53%. Ablating trace selection (Phase 1), diagnostic slicing (Phase 2), or target attribution (Phase 3) individually reduced mean success by 7.00, 7.50, and 9.33 percentage points, respectively. A 2×2 comparison of trace context and edit targets further showed that combining slicing with attribution outperformed either alone. All three phases therefore contribute, and slicing and attribution work best together.
  • Generalization to modern coding agents. On 294 held-out SWE-bench Pro tasks across 11 repositories, TracePrism increased the success rate of SWE-agent with GPT-5.6-Luna from 80.95% to 85.03%, compared with 82.31% for GEPA. In a separate transfer experiment, policies optimized on one repository outperformed both baselines on each of three unseen repositories. TracePrism therefore generalizes to a modern coding-agent setting, as demonstrated on SWE-bench Pro.
  • Representative selection at scale. With both methods selecting 22 traces from the same 303-trace training pool, representative selection achieved 53.83% mean success versus 47.67% for random sampling across three runs, at a slightly lower mean optimization cost. Representative selection therefore outperforms random selection at scale.

In summary, these results show that each phase of TracePrism contributes to its gains, that the method generalizes to a modern coding-agent setting on SWE-bench Pro, and that representative selection outperforms random sampling at scale. We thank the Area Chair and all reviewers again for their consideration.

Regards from the authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.