- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposed a new debug method, TracwACE, collection-level agent-trace debugging to diagnoise issues in agentic tasks. The method first starts with trace preprocessing/normalize including convert traces into canonical format, compact repeated payload, and organize them into episode chunk. And taxonomy-conditioned LLM scanners detect and localize issues in each chunk. Then TraceACE groups the issues by a deterministic signature, coarsens the groups and let a LLM merge all isses that can be solved by one fix into one cluster. Finally a tool-using agent investigates one representative trace per cluster. An evidence gate checks every quotation the agent cites, and a validator replays the agent's hypothesis on other cluster members. The authors evaluate TraceACE against seven baselines with eight LLM backbones on TRAIL and AgentErrorBench. They report large gains on TRAIL and mixed results on AgentErrorBench.
Critical strengths
-
The experiments are extensive The experiments compare lots of similair debug methods, including generic judges (single-pass LLM, ReAct) and the main recent trace-diagnosis methods (CORRECT, AgentDebug, AgentRx, AgentDebugX, Agentic CLEAR). The appendix also adds Self-Refine, Plan-and-Solve and CRITIC. The authors run every method under eight models and variuos of benchmarks. The experiments results show that traceACE leads GAIA F1 localization and joint accuracy under all eight backbones, and leads SWE joint accuracy under seven. It seems like TraceACE has the best score under five of eight backbones in both environments.
-
The engineering method seems solid The paper uses a complex but careful/auditable workflow to better debug agent traces. Including trace normalization, verbatim span IDs, store ground truth outside the agent's workspace, filtered issues kept with a status and reason, merge-only clustering and a mechanical quotation check all seems good engineering choices.
Critical weaknesses
-
The paper lists clustering and explanation as its main contributions, but the experiments do not isolate their effect. The authors score TRAIL only on the output of the detection stage, so the +24.2 GAIA would be the same without clustering or explanation. The paper also has no same-fix labels and no collection-level metric, so it never directly tests whether the issues in a cluster share one fix. Also, the paper's own check suggests that many clusters do not share one explanation. TraceACE writes one explanation per cluster from the representative trace, then checks whether it also fits other traces in the same cluster. On the 30-trace TRAIL cohort, the authors ran this check on 22 traces. The explanation fit 9 of them, fit 4 partly, and did not fit the remaining 9 (Table D.6). For 13 of the 22 traces, the cluster's explanation does not fully apply.
The paper also claims "two timeouts may need different fixes" (L131). However, a seed group requires an identical signature (Eq. B.7) of (category, span kind, tool, error token). The merge pass may not split a seed group (Eq. B.8). The clustering step considers fixes only when it merges groups. It cannot separate issues that share a symptom but need different fixes.
- The novelty mainly comes from combining existing components Each part of TraceACE seems to use method from on the existing paper:
- AgentDebug already splits error analysis by taxonomy module.
- AgentDebugX already produces evidence-backed diagnoses with cross-examination and a suggested repair.
- Agentic CLEAR and CORRECT already summarize recurring issues across traces.
Questions for authors
-
How are AEB predictions made? AEB needs one answer per trace. TraceACE investigates one representative trace per cluster and checks 2–5 other members, so many traces are never examined in this stage. Where do their AEB answers come from? If they come from the detection stage, clustering and explanation change only a few answers, and Table 3 cannot show their effect. Also, a trace can belong to several clusters (140 clusters for 100 ALFWorld traces). Which cluster decides its answer?
-
Please addresses the problem in weakness.
6 comments
[W1] Isolating Clustering and Explanation, Same-Fix Labels, and the 13 of 22 Verdicts [part 2/2]
Our direct answer on the 22 verdicts: Your reading of Table D.6 is correct. For 13 of the 22 checked traces the cluster's explanation does not fully apply, and the first-pass clusters of that cohort over-merge. What the submission did not show is what happens next: the 13 verdicts sit in four of the five clusters, all four went to re-diagnosis, three were split, and the eight clusters of the cohort became eleven mechanism groups. We also now show that the verdicts track human same-fix labels and predict whether a repair transfers.
1. The five clusters behind the 22 verdicts (App. E.5; Table E.4, panel c):
| Cluster | Verdicts | Confirmed / partial / refuted | Status | Re-diagnosis | Regrouped | Unassigned |
|---|---|---|---|---|---|---|
| A | 2 | 2/0/0 | confirmed | not triggered | — | — |
| B | 5 | 3/1/1 | mixed | one group, revised explanation | 2 | 0 |
| C | 5 | 2/1/2 | mixed | split by mechanism | 2 | 1 |
| D | 5 | 2/1/2 | mixed | split by mechanism | 3 | 0 |
| E | 5 | 0/1/4 | refuted | split by mechanism | 3 | 2 |
| All | 22 | 9/4/9 | 10 | 3 |
Cluster B kept a single group whose revised explanation covers its two non-confirming members. In C, D, and E, eight non-confirming members form three second-mechanism groups and three are left unassigned.
2. Is over-merging specific to that small cohort? No (Table E.4, panel b):
| Cohort | Clusters | Single-trace | Confirmed | Mixed | Refuted |
|---|---|---|---|---|---|
| ALFWorld | 140 | 101 | 17 | 17 | 5 |
| WebShop | 45 | 27 | 13 | 3 | 2 |
| Both | 185 | 128 | 30 | 20 | 7 |
So 27 of the 57 multi-trace clusters (47.4%) are not confirmed at the first pass. Re-diagnosis revised 15 of them, split 9 (11 new groups), and in 3 only moved issues to unassigned. Scored against the blinded same-fix partition, this raises B-cubed precision from 0.872 to 0.967 on ALFWorld and from 0.832 to 0.877 on WebShop, at nearly unchanged recall (0.639 and 0.789). Over the issues in multi-trace clusters alone, precision rises from 0.824 to 0.954 and from 0.798 to 0.851.
3. Do the verdicts mean anything? Two checks.
- Against human labels (99 verdicts on the AEB cohorts; Table E.4, panel a). A confirmed member shares the representative's fix group in 63 of 67 cases (94.0%), a partial one in 14 of 21 (66.7%), a refuted one in 4 of 11 (36.4%).
- Against executed repairs (23 clusters; Table E.6, Figure E.2). Among the 117 members a fix was not written from, it transfers to 91.7% in the 11 confirmed clusters, 66.7% in the 8 mixed, and 20.0% in the 4 refuted ones ( between status and cluster rate; permutation ). In the ten largest clusters, it transfers to 19 of 20 members with a confirmed verdict (95.0%) and 2 of 7 with a refuted one (28.6%).
A non-confirming verdict is thus a useful signal that the cluster's explanation, and its fix, will not carry over. The 13 of 22 show the validator at work on clusters that were too coarse.
4. What we concede.
- First-pass clusters are proposals. About half of the multi-trace clusters need revision or splitting (App. E.5).
- Most members of a large cluster are never checked. The validator samples at most five members, and a cluster whose sampled members all confirm is never re-diagnosed; §3.3 now says that a confirmed status refers to the sampled members only. In the repair study the fix still transfers to 39 of the 51 unsampled members (76.5%), which is lower than for confirmed members. Re-diagnosis itself is not limited to the sample: its assignment pass reads every issue of the cluster, and 13 of the 25 issues it moved on ALFWorld (2 of 5 on WebShop) lie on traces the validator never sampled. Against the labels, 114 of the 125 issues on never-sampled members of confirmed clusters share the representative's fix group (91.2%); the other 11 are heterogeneity that sampling missed.
- Single-trace clusters cannot be validated at all: 128 of the 185 clusters of the AEB cohorts. They hold 27.2% (ALFWorld) and 17.2% (WebShop) of the issues.
- Re-diagnosis on TRAIL is unchecked. Its regrouping was compared to labels and repairs only on the AEB and repair cohorts.
- The validator is weak against plausible wrong hypotheses. Of 73 near-miss verdicts verified as negatives it confirms 6.8% and refutes only 64.4% (App. E.2).
Summary of manuscript changes for W1, part 2 (blue in the revised PDF):
| Location | Change |
|---|---|
| App. E.5; Table E.4, panels b and c | cluster status and re-diagnosis outcomes on AEB; the five held-out clusters behind the 22 verdicts |
| App. D.7, App. C.7 | pointers from the 22 verdicts to their breakdown and to the near-miss controls |
| App. B.4, App. E.5 | re-diagnosis enabled in every reported run; what its assignment pass reads; support by issues |
| Table E.4 (panel a), Table E.6 | verdicts against same-fix labels and against repair transfer |
[W2] "Two Timeouts May Need Different Fixes" versus Merge-Only Clustering [part 1/1]
Thank you, you are right: the sentence overstated what the clustering stage does. Our direct answer: A seed group requires an identical signature (Eq. B.7), and the merge pass can only join groups (Eq. B.8), so the clustering stage cannot separate two issues that share a symptom but need different fixes. The only component that can is the validation-triggered re-diagnosis, which the submission mentioned as "optional" and never evaluated. We corrected the text in §2, §3.3 and App. B.3, and now quantify the problem and how much of it re-diagnosis recovers.
1. How large is the problem? In the blinded same-fix annotation (App. E.4), 157 issue pairs share a signature (98 on ALFWorld, 59 on WebShop). The annotators placed 21 of them (13.4%; 10 and 11) in different fix groups. These are your "two timeouts": same category, span kind, tool, and error token, different fix. Clustering keeps all of them together. The merge pass adds its own errors: 11 of its 160 merge decisions on ALFWorld and 3 of 80 on WebShop join seed groups of different fix groups.
2. What can separate them: validation, then re-diagnosis (App. B.4, App. E.5). When the representative's hypothesis is replayed on other members and a verdict is not confirmed, the cluster becomes mixed or refuted and is regrouped by observed mechanism. Re-diagnosis is behind a configuration switch, hence "optional"; App. B.4 now states that it is enabled in every reported run.
| ALFWorld | WebShop | |
|---|---|---|
| Multi-trace clusters | 39 | 18 |
| Mixed or refuted at the first pass | 22 | 5 |
| Re-diagnosis: revised / split / unassigned only | 12 / 7 / 3 | 3 / 2 / 0 |
| New mechanism groups | 9 | 2 |
| B-cubed precision, before and after | 0.872 and 0.967 | 0.832 and 0.877 |
| B-cubed precision, multi-trace clusters only | 0.824 and 0.954 | 0.798 and 0.851 |
| B-cubed recall, before and after | 0.656 and 0.639 | 0.799 and 0.789 |
| Pairwise precision, before and after | 0.73 and 0.94 | 0.77 and 0.80 |
Scores are against the blinded same-fix partition (Table E.3).
How a few checked members yield a partition of all issues (App. E.5). Re-diagnosis proposes mechanism groups from the recorded verdicts; an assignment pass then reads every issue of the cluster with its cited span and assigns it to a group or to unassigned, issue by issue. The 22 and 5 re-diagnosed clusters hold 159 and 34 issues in 133 and 31 cluster–trace pairs; the validator had sampled 41 and 13 of these. The pass moved 25 and 5 issues, 13 and 2 of them on never-sampled traces, and moved 25 of 28 and 5 of 5 issues whose reference fix group differs from the representative's. All other clusters keep every issue.
Executed repairs agree: of 17 members a cluster's first fix did not resolve, re-diagnosis placed 12 in 5 second-mechanism groups, and one further patch per group transfers to 9 of the 12 (75.0%; App. E.6).
3. What this does not solve.
- Unsampled members. The validator samples at most five members, and a cluster whose sampled members all confirm is never re-diagnosed. A same-signature pair with different fixes inside such a cluster stays merged: by the labels, 114 of the 125 issues on never-sampled members of confirmed clusters share the representative's fix group (91.2%); 11 do not.
- Single-trace clusters. 101 of the 140 ALFWorld clusters and 27 of the 45 WebShop clusters hold one trace and cannot be validated; they hold 27.2% and 17.2% of the issues.
- Near-misses. The validator refutes only 64.4% of verified near-miss hypotheses (App. E.2).
- A confirmed verdict is not a guarantee. 4 of the 67 confirmed members belong to another fix group than the representative (Table E.4, panel a).
4. Why the merge pass stays merge-only. We considered letting the model split seed groups and decided against it (App. B.3): the deterministic groups must survive a model failure, and splitting a signature group requires reading the traces, which only investigation and validation do. The cost is the 13.4% above.
5. The corrected text. §3.3 now reads: "Clustering thus cannot separate issues that share a signature (21 of 157 such pairs, 13.4%, need different fixes in our blinded annotation; Appendix E.4); that is left to the re-diagnosis described next." The L131 sentence stays as a statement of the target, now followed by which stage can act on it.
Summary of manuscript changes for W2 (blue in the revised PDF):
| Location | Change |
|---|---|
| §2, §3.3, App. B.3 | clustering can only join issues; share of identical-signature pairs that need different fixes; reasons for the merge-only constraint |
| App. B.4, App. E.5, Table E.4 (panel b) | re-diagnosis enabled in all runs; how often it revises, splits, or leaves issues unassigned; what its assignment pass reads |
| Table E.3, §6 | partition quality before and after re-diagnosis; the merge-only limit listed in §6 |
[W1] Isolating Clustering and Explanation, Same-Fix Labels, and the 13 of 22 Verdicts [part 1/2]
Thank you, all three points are correct. Our direct answer: TRAIL is scored on the detection output only, so the +24.2 Joint points on GAIA would be identical without clustering or explanation, and the revised paper says so. The revision adds what was missing: a blinded same-fix annotation with clustering metrics, an executed-repair study, and an analysis-only readout that isolates the explanation stage on AEB (+5.04 points pooled, 95% interval [+2.11, +8.62]). Clustering itself does not raise root accuracy: over five runs it is equivalent to per-trace investigation within a preset margin of ±3 points, and we withdrew the claim of a gain. Part 2/2 covers the 13 of 22 verdicts.
1. What TRAIL measures. §1 now states after the contribution list: "TRAIL detection scores measure contributions 1 and 2 only; contributions 3 and 4 are evaluated separately, on AgentErrorBench, against blinded same-fix labels, and by executed repairs (Sections 4.5 and 4.6)."
2. Isolating the explanation stage on AEB (App. D.9 and E.7). A new arm, the analysis-only readout, answers each trace from its localized issues alone. Five runs per arm on frozen analysis outputs:
3. Same-fix labels and a collection-level metric (App. E.4). Two annotators who never saw TraceACE's grouping sorted all 534 issues of the two AEB runs (371 and 163) into groups that one concrete change would fix; κ over all issue pairs is 0.81 and 0.74, and a third annotator adjudicated (58 and 27 reference fix groups). B-cubed precision / recall / F1 against this partition (Table E.3):
This is the direct test of whether issues in a cluster share one fix. The answer is "mostly, not always": first-pass precision is 0.872 and 0.832. This flatters multi-trace clusters: 101 and 26 clusters hold a single issue and are pure by definition. Over the issues in multi-trace clusters it is 0.824 and 0.798, and pairwise precision is 0.73 and 0.77. Recall is moderate: 140 and 45 clusters stand against 58 and 27 reference groups.
4. Executed repairs (App. E.6). On a re-runnable cohort we implemented the fix of ten clusters and re-ran all 95 cluster–member pairs three times per arm. The cluster's own fix removes the target issue in 76.5% of the 285 re-runs, against 16.5% with no change, and transfers to 64 of the 85 members it was not written from (75.3%); over 23 repaired clusters, to 91.7% in confirmed and 20.0% in refuted clusters. Applied unchanged to the CLEAR-style or LLM-only groups around the same traces, the same ten patches reach 38.2% and 45.5% of the other members. Together, they raise the agent's success by 7.7 points over four task sets, two of them held out; against the two groupings an end-to-end lead appears only when the patches are generated automatically from each workflow's own output (+2.9 and +2.8 points), and it falls to +1.4 and +1.3, not established, once every grouping gets an investigated report (Tables E.6–E.9; W3, part 2/2).
5. What this does not show. One backbone; 23 repaired clusters of one ReAct-style agent; the TRAIL agents could not be re-run, so on TRAIL the collection-level stages are evaluated only through validator verdicts (part 2/2).
Summary of manuscript changes for W1, part 1 (blue in the revised PDF):