- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TracePrism improves agent component instructions using a four-stage diagnosis pipeline. It profiles and clusters failed trajectories, selects representative cases under a deep-diagnosis budget, constructs backward slices following inferred data/control dependencies, maps plausible failure causes to editable components, and applies scoped instruction revisions. On VeruSAGE-Bench, WebArena, and HotpotQA, it obtains the highest mean performance among the evaluated methods, including a VeruSAGE success increase from 46.2% to 58.8% across three independent optimization runs. Additional experiments examine optimizer-side cost as the failure corpus grows, selection and diagnostic-context ablations, and agent/step attribution on Who&When.
Critical strengths
- Meaningful end-to-end improvements across diverse tasks.
TracePrism demonstrates consistent improvements across formal verification, browser interaction, and multi-hop QA, with three independent optimization runs in the main evaluation. On VeruSAGE, it improves performance by 12.6 percentage points over the base agent and 11.0 points over the strongest evaluated baseline, with additional gains on WebArena and HotpotQA. These results demonstrate the effectiveness of the framework across different task structures rather than a single specialized workflow.
- A well-motivated framework for identifying actionable optimization opportunities.
The paper presents a coherent approach to analyzing agent failures by separating failure selection, upstream evidence tracing, and identification of the components responsible for errors. This addresses an important challenge in modular agents: the component where a failure becomes visible is not necessarily the one responsible for the underlying decision. The framework provides a practical way to identify where optimization efforts should be focused.
Critical weaknesses
- Limited evaluation on modern long-horizon agents.
The main experiments use o4-mini and GPT-4o backbones, which may not fully reflect the capabilities of stronger, more recent agents. Additionally, HotpotQA uses a relatively simple four-stage workflow, while WebArena has only one editable instruction, making it difficult to evaluate the framework’s ability to identify the right component to optimize. Experiments with stronger backbones and more complex, long-horizon coding or terminal tasks would help demonstrate the framework’s generalizability to modern agentic workflows.
- Limited evidence supporting the effectiveness of individual components.
The diagnostic ablations are relatively small, using only ten failed trajectories and a 50-task evaluation subset with a single run per setting. Notably, both Current-context and Full-trace Repair achieve 54% success, compared to 56% for the full pipeline—a difference of just one task. These results make it difficult to determine whether the proposed diagnosis mechanism provides a meaningful advantage over simpler alternatives. Larger-scale ablations with multiple runs would help establish the contribution of each component.
Questions for authors
- Budget comparability: How would the main results change if all methods were given the same eligible traces, editable components, and total optimization budget, including profiling, clustering, diagnosis, and policy revision?
- Effectiveness of the diagnosis mechanism: Does the 54% vs. 56% performance difference hold on larger test sets or across multiple runs? Could the authors separately ablate evidence slicing and target attribution while keeping trace selection and revision procedures fixed?
- Generalization to long-horizon agents: Have the authors evaluated TracePrism on more complex coding or terminal benchmarks, such as Terminal-Bench or SWE-bench, using stronger, more recent model backbones? Such results would help establish whether the improvements generalize to modern long-horizon agent workflows.
5 comments
Response to Reviewer Ffm9 (2/2)
These results establish two clear conclusions:
- The diagnostic advantage is supported by a larger, repeated evaluation: In our new three-run evaluation on 200 test tasks, TracePrism achieves a mean success rate of 52.83 ± 1.53%, outperforming
Full Trace + Manifestationby 6.67 percentage points, while reducing optimizer cost by 61.4%. These results strengthen the empirical support for the effectiveness and optimizer-side efficiency of the proposed diagnostic mechanisms. - Independent contributions of Evidence Slicing and Target Attribution: Removing either stage while keeping trace selection and revision procedures fixed severely degrades performance. Without evidence slicing (
Full Trace + Attribution, -7.50% SR), long-horizon irrelevant steps and retry noise overwhelm raw-trace attribution and inflate token overhead; without target attribution (Sliced Trace + Manifestation, -9.33% SR), revising the error-manifestation site rather than the upstream responsible component leaves underlying decision flaws unresolved.
Q1: Budget Comparability & End-to-End Cost Accounting
To address budget comparability, we conducted an additional controlled comparison between TracePrism and GEPA on HotpotQA, using 200 training examples, 50 validation examples, and 300 held-out test examples. Both methods were given the same pool of eligible traces, identical editable components, and the same $2.00 total optimization budget cap, covering all optimization-stage costs, including trace collection, profiling, diagnosis, policy revision, and candidate evaluation.
We chose HotpotQA for this experiment because multi-round optimization requires repeated agent executions for trace collection and candidate evaluation, making budget-matched experiments on VeruSAGE substantially more time-consuming. Even on HotpotQA, GEPA required 6 hours and 9 minutes to complete its run.
| Method | Optimization Rounds | Agent Calls | Best Validation Score | Test Score |
|---|---|---|---|---|
| GEPA | 20 | 1,240 | 56.0% | 61.0% (183/300) |
| TracePrism | 1 | 250 | 58.0% | 64.0% (192/300) |
GEPA's final candidate was selected based on the highest validation score from its Pareto frontier. Under the same $2.00 optimization budget, TracePrism outperforms GEPA by 3.0 percentage points on the held-out test set, solving 9 additional tasks while requiring 79.8% fewer agent calls. These results demonstrate that TracePrism maintains its performance advantage when eligible traces, editable components, and total optimization budget are matched.
We have updated the revised manuscript with all clarifications and new experimental results above, highlighted in blue for ease of review. Thank you again for your time and constructive feedback.
Follow-up on Our Rebuttal
Dear Reviewer,
We hope this message finds you well. As the discussion period draws to a close, we wanted to follow up on our response to your review. If any of your concerns remain or if anything in our response is unclear, we’d be happy to clarify. Your insights are invaluable to us.
Thank you for taking the time to review our paper.
Best regards,
The Authors
Response to Reviewer Ffm9 (1/2)
We sincerely thank Reviewer Ffm9 for the insightful feedback and constructive suggestions. Below, we address W1–W2 and Q1–Q3 with new experimental results on SWE-bench Pro using modern model backbones, alongside our new 3-seed independent ablations on VeruSAGE-Bench.
W1 & Q3: Generalization to modern long-horizon benchmarks and agents
To evaluate whether TracePrism generalizes to complex, long-horizon software engineering agents powered by stronger, more recent backbones, we conducted two complementary sets of experiments on SWE-bench Pro and compared TracePrism against GEPA, a prominent agent optimization baseline. Both evaluations use
SWE-agentas the base agent,GPT-5.6-Lunaas the target model, andGPT-6-Astraas the optimizer across distinct data splits and reasoning regimes.1. Stratified Full-Benchmark Evaluation (
GPT-5.6-Luna,maxreasoning effort)First, we partitioned the full SWE-bench Pro benchmark using a repository-stratified split into 292 training tasks, 145 validation tasks, and 294 held-out test tasks across 11 repositories:
NodeBB/NodeBBansible/ansibleelement-hq/element-webflipt-io/fliptfuture-architect/vulsgravitational/teleportinternetarchive/openlibrarynavidrome/navidromeprotonmail/webclientsqutebrowser/qutebrowsertutao/tutanotaEven the base agent already solves 80.95% of tasks, TracePrism improves the overall success rate by +4.08 percentage points over the base agent (+12 solved tasks) and +2.72 percentage points over GEPA (+8 solved tasks), achieving the highest or tied-highest score on 7 of the 11 repositories.
2. Cross-Repository Transfer Evaluation (
GPT-5.6-Luna,lowreasoning effort):Second, to test cross-repository generalization under a tighter reasoning budget, we optimized policies solely on failed trajectories from
qutebrowserand evaluated them on 237 held-out test tasks across three unseen repositories:TracePrism improves the overall success rate by +4.22 percentage points over the base agent (+10 solved tasks) and +3.80 percentage points over GEPA (+9 solved tasks), outperforming both baselines across every evaluated repository.
W2 & Q2: Larger-scale 3-seed independent ablations
To verify whether the diagnostic gains hold on larger test sets and across multiple runs, and to separately ablate evidence slicing (Phase 2) and target attribution (Phase 3), we conducted new 3-seed ablations on VeruSAGE-Bench (50-task training pool, 200 test tasks; target agent:
gpt-o4-mini). Due to API quota limits, we unified the optimizer model across all settings in these new experiments toGPT-6.1-solwith high thinking effort, keeping the optimizer configuration consistent.