acceptodds
Under review as a conference paper at ICLR 2027

TraceACE: Analysis, Clustering, and Explanation of Failures in LLM-Agent Traces

Abstract

Debugging an LLM agent from its final answer is underdetermined: the answer does not reveal which span introduced an error, whether failures across traces share one fix, or whether a fluent root-cause narrative matches recorded execution. This paper presents TRACEACE, a collection-level workflow that turns heterogeneous observability traces into localized issues, interpretable same-fix clusters, evidence-constrained root-cause hypotheses, and an engineering report. Canonicalization preserves source span identifiers; taxonomy-conditioned scanners run alongside optional rule-based anchors and a structural channel; transparent signatures and a conservative merge form same-fix clusters, each investigated via one representative; and every hypothesis quotation is checked against the workspace before replay on secondary traces, escalating on disagreement. Across TRAIL and AgentErrorBench with eight LLM backbones and seven baselines, TRACEACE attains the highest joint accuracy on TRAIL GAIA under every backbone, with a 24.2-point mean gain over the strongest baseline, and on TRAIL SWE under seven of eight; on critical root-cause attribution it matches the strongest per-trace attributors. Ablations show the largest losses when specialized scanners are replaced by a unified one (up to 3.3 joint points) or cross-trace validation is removed (about 4 points of root accuracy). TRACEACE uses less than half the per-trace tokens of AgentDebug, and cluster-level investigation raises root accuracy over per-trace investigation on both AgentErrorBench environments (16.2 vs. 14.9 and 20.4 vs. 18.4). These are single-run point estimates; same-fix gold labels and interventional validation remain unavailable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, Observability Traces, Failure Localization, Root-Cause Analysis, Clustering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YUXjOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed a new debug method, TracwACE, collection-level agent-trace debugging to diagnoise issues in agentic tasks. The method first starts with trace preprocessing/normalize including convert traces into canonical format, compact repeated payload, and organize them into episode chunk. And taxonomy-conditioned LLM scanners detect and localize issues in each chunk. Then TraceACE groups the issues by a deterministic signature, coarsens the groups and let a LLM merge all isses that can be solved by one fix into one cluster. Finally a tool-using agent investigates one representative trace per cluster. An evidence gate checks every quotation the agent cites, and a validator replays the agent's hypothesis on other cluster members. The authors evaluate TraceACE against seven baselines with eight LLM backbones on TRAIL and AgentErrorBench. They report large gains on TRAIL and mixed results on AgentErrorBench.

Critical strengths
  1. The experiments are extensive The experiments compare lots of similair debug methods, including generic judges (single-pass LLM, ReAct) and the main recent trace-diagnosis methods (CORRECT, AgentDebug, AgentRx, AgentDebugX, Agentic CLEAR). The appendix also adds Self-Refine, Plan-and-Solve and CRITIC. The authors run every method under eight models and variuos of benchmarks. The experiments results show that traceACE leads GAIA F1 localization and joint accuracy under all eight backbones, and leads SWE joint accuracy under seven. It seems like TraceACE has the best score under five of eight backbones in both environments.

  2. The engineering method seems solid The paper uses a complex but careful/auditable workflow to better debug agent traces. Including trace normalization, verbatim span IDs, store ground truth outside the agent's workspace, filtered issues kept with a status and reason, merge-only clustering and a mechanical quotation check all seems good engineering choices.

Critical weaknesses
  1. The paper lists clustering and explanation as its main contributions, but the experiments do not isolate their effect. The authors score TRAIL only on the output of the detection stage, so the +24.2 GAIA would be the same without clustering or explanation. The paper also has no same-fix labels and no collection-level metric, so it never directly tests whether the issues in a cluster share one fix. Also, the paper's own check suggests that many clusters do not share one explanation. TraceACE writes one explanation per cluster from the representative trace, then checks whether it also fits other traces in the same cluster. On the 30-trace TRAIL cohort, the authors ran this check on 22 traces. The explanation fit 9 of them, fit 4 partly, and did not fit the remaining 9 (Table D.6). For 13 of the 22 traces, the cluster's explanation does not fully apply.

The paper also claims "two timeouts may need different fixes" (L131). However, a seed group requires an identical signature (Eq. B.7) of (category, span kind, tool, error token). The merge pass may not split a seed group (Eq. B.8). The clustering step considers fixes only when it merges groups. It cannot separate issues that share a symptom but need different fixes.

  1. The novelty mainly comes from combining existing components Each part of TraceACE seems to use method from on the existing paper:
  • AgentDebug already splits error analysis by taxonomy module.
  • AgentDebugX already produces evidence-backed diagnoses with cross-examination and a suggested repair.
  • Agentic CLEAR and CORRECT already summarize recurring issues across traces.
Questions for authors
  1. How are AEB predictions made? AEB needs one answer per trace. TraceACE investigates one representative trace per cluster and checks 2–5 other members, so many traces are never examined in this stage. Where do their AEB answers come from? If they come from the detection stage, clustering and explanation change only a few answers, and Table 3 cannot show their effect. Also, a trace can belong to several clusters (140 clusters for 100 ALFWorld traces). Which cluster decides its answer?

  2. Please addresses the problem in weakness.

6 comments
AuthorsOpenReview ↗

[W1] Isolating Clustering and Explanation, Same-Fix Labels, and the 13 of 22 Verdicts [part 1/2]

Thank you, all three points are correct. Our direct answer: TRAIL is scored on the detection output only, so the +24.2 Joint points on GAIA would be identical without clustering or explanation, and the revised paper says so. The revision adds what was missing: a blinded same-fix annotation with clustering metrics, an executed-repair study, and an analysis-only readout that isolates the explanation stage on AEB (+5.04 points pooled, 95% interval [+2.11, +8.62]). Clustering itself does not raise root accuracy: over five runs it is equivalent to per-trace investigation within a preset margin of ±3\pm3 points, and we withdrew the claim of a gain. Part 2/2 covers the 13 of 22 verdicts.

1. What TRAIL measures. §1 now states after the contribution list: "TRAIL detection scores measure contributions 1 and 2 only; contributions 3 and 4 are evaluated separately, on AgentErrorBench, against blinded same-fix labels, and by executed repairs (Sections 4.5 and 4.6)."

2. Isolating the explanation stage on AEB (App. D.9 and E.7). A new arm, the analysis-only readout, answers each trace from its localized issues alone. Five runs per arm on frozen analysis outputs:

VariantALFWorld AllWebShop AllPost-analysis tokens per trace
Analysis-only readout10.8±1.010.8\pm1.014.3±1.414.3\pm1.421.6k / 24.9k
Per-trace RCA15.4±0.715.4\pm0.719.6±1.119.6\pm1.1503.1k / 681.7k
TraceACE15.7±0.715.7\pm0.719.6±1.119.6\pm1.11,537.2k / 1,604.8k
  • Explanation stage as a whole. TraceACE minus the analysis-only readout is +4.86 points on ALFWorld [+1.08, +9.46], +5.31 on WebShop [+0.41, +11.02], and +5.04 pooled [+2.11, +8.62].
  • Clustering over per-trace investigation. The pooled difference is +0.16 points [−2.11, +2.44], and −0.33 [−2.76, +2.11] against a per-trace arm with a matched token budget. Both intervals lie inside the ±3\pm3-point margin: equivalence, not a gain. Per cohort, equivalence is not established.

3. Same-fix labels and a collection-level metric (App. E.4). Two annotators who never saw TraceACE's grouping sorted all 534 issues of the two AEB runs (371 and 163) into groups that one concrete change would fix; κ\kappa over all issue pairs is 0.81 and 0.74, and a third annotator adjudicated (58 and 27 reference fix groups). B-cubed precision / recall / F1 against this partition (Table E.3):

GroupingALFWorldWebShop
Signature only0.975 / 0.190 / 0.3180.957 / 0.221 / 0.359
CLEAR-style issue synthesis0.532 / 0.678 / 0.5960.500 / 0.733 / 0.595
TraceACE0.872 / 0.656 / 0.7480.832 / 0.799 / 0.815
TraceACE + re-diagnosis0.967 / 0.639 / 0.7700.877 / 0.789 / 0.830

This is the direct test of whether issues in a cluster share one fix. The answer is "mostly, not always": first-pass precision is 0.872 and 0.832. This flatters multi-trace clusters: 101 and 26 clusters hold a single issue and are pure by definition. Over the issues in multi-trace clusters it is 0.824 and 0.798, and pairwise precision is 0.73 and 0.77. Recall is moderate: 140 and 45 clusters stand against 58 and 27 reference groups.

  • Signatures alone are not fix groups. 21 of the 157 identical-signature issue pairs (13.4%) were placed in different fix groups (see W2).

4. Executed repairs (App. E.6). On a re-runnable cohort we implemented the fix of ten clusters and re-ran all 95 cluster–member pairs three times per arm. The cluster's own fix removes the target issue in 76.5% of the 285 re-runs, against 16.5% with no change, and transfers to 64 of the 85 members it was not written from (75.3%); over 23 repaired clusters, to 91.7% in confirmed and 20.0% in refuted clusters. Applied unchanged to the CLEAR-style or LLM-only groups around the same traces, the same ten patches reach 38.2% and 45.5% of the other members. Together, they raise the agent's success by 7.7 points over four task sets, two of them held out; against the two groupings an end-to-end lead appears only when the patches are generated automatically from each workflow's own output (+2.9 and +2.8 points), and it falls to +1.4 and +1.3, not established, once every grouping gets an investigated report (Tables E.6–E.9; W3, part 2/2).

5. What this does not show. One backbone; 23 repaired clusters of one ReAct-style agent; the TRAIL agents could not be re-run, so on TRAIL the collection-level stages are evaluated only through validator verdicts (part 2/2).

Summary of manuscript changes for W1, part 1 (blue in the revised PDF):

LocationChange
Abstract, §1, §6TRAIL scores attributed to the analysis stage; clustering and explanation evaluated separately
§4.5, App. D.9, Table D.8analysis-only and budget-matched arms, five runs, preset margin; accuracy-gain claim withdrawn
§4.6, Table 4, App. E.4, App. E.6same-fix annotation with B-cubed and pairwise scores; executed repairs
AuthorsOpenReview ↗

[W1] Isolating Clustering and Explanation, Same-Fix Labels, and the 13 of 22 Verdicts [part 2/2]

Our direct answer on the 22 verdicts: Your reading of Table D.6 is correct. For 13 of the 22 checked traces the cluster's explanation does not fully apply, and the first-pass clusters of that cohort over-merge. What the submission did not show is what happens next: the 13 verdicts sit in four of the five clusters, all four went to re-diagnosis, three were split, and the eight clusters of the cohort became eleven mechanism groups. We also now show that the verdicts track human same-fix labels and predict whether a repair transfers.

1. The five clusters behind the 22 verdicts (App. E.5; Table E.4, panel c):

ClusterVerdictsConfirmed / partial / refutedStatusRe-diagnosisRegroupedUnassigned
A22/0/0confirmednot triggered——
B53/1/1mixedone group, revised explanation20
C52/1/2mixedsplit by mechanism21
D52/1/2mixedsplit by mechanism30
E50/1/4refutedsplit by mechanism32
All229/4/9103

Cluster B kept a single group whose revised explanation covers its two non-confirming members. In C, D, and E, eight non-confirming members form three second-mechanism groups and three are left unassigned.

2. Is over-merging specific to that small cohort? No (Table E.4, panel b):

CohortClustersSingle-traceConfirmedMixedRefuted
ALFWorld14010117175
WebShop45271332
Both18512830207

So 27 of the 57 multi-trace clusters (47.4%) are not confirmed at the first pass. Re-diagnosis revised 15 of them, split 9 (11 new groups), and in 3 only moved issues to unassigned. Scored against the blinded same-fix partition, this raises B-cubed precision from 0.872 to 0.967 on ALFWorld and from 0.832 to 0.877 on WebShop, at nearly unchanged recall (0.639 and 0.789). Over the issues in multi-trace clusters alone, precision rises from 0.824 to 0.954 and from 0.798 to 0.851.

3. Do the verdicts mean anything? Two checks.

  • Against human labels (99 verdicts on the AEB cohorts; Table E.4, panel a). A confirmed member shares the representative's fix group in 63 of 67 cases (94.0%), a partial one in 14 of 21 (66.7%), a refuted one in 4 of 11 (36.4%).
  • Against executed repairs (23 clusters; Table E.6, Figure E.2). Among the 117 members a fix was not written from, it transfers to 91.7% in the 11 confirmed clusters, 66.7% in the 8 mixed, and 20.0% in the 4 refuted ones (r=0.89r=0.89 between status and cluster rate; permutation p<0.001p < 0.001). In the ten largest clusters, it transfers to 19 of 20 members with a confirmed verdict (95.0%) and 2 of 7 with a refuted one (28.6%).

A non-confirming verdict is thus a useful signal that the cluster's explanation, and its fix, will not carry over. The 13 of 22 show the validator at work on clusters that were too coarse.

4. What we concede.

  • First-pass clusters are proposals. About half of the multi-trace clusters need revision or splitting (App. E.5).
  • Most members of a large cluster are never checked. The validator samples at most five members, and a cluster whose sampled members all confirm is never re-diagnosed; §3.3 now says that a confirmed status refers to the sampled members only. In the repair study the fix still transfers to 39 of the 51 unsampled members (76.5%), which is lower than for confirmed members. Re-diagnosis itself is not limited to the sample: its assignment pass reads every issue of the cluster, and 13 of the 25 issues it moved on ALFWorld (2 of 5 on WebShop) lie on traces the validator never sampled. Against the labels, 114 of the 125 issues on never-sampled members of confirmed clusters share the representative's fix group (91.2%); the other 11 are heterogeneity that sampling missed.
  • Single-trace clusters cannot be validated at all: 128 of the 185 clusters of the AEB cohorts. They hold 27.2% (ALFWorld) and 17.2% (WebShop) of the issues.
  • Re-diagnosis on TRAIL is unchecked. Its regrouping was compared to labels and repairs only on the AEB and repair cohorts.
  • The validator is weak against plausible wrong hypotheses. Of 73 near-miss verdicts verified as negatives it confirms 6.8% and refutes only 64.4% (App. E.2).

Summary of manuscript changes for W1, part 2 (blue in the revised PDF):

LocationChange
App. E.5; Table E.4, panels b and ccluster status and re-diagnosis outcomes on AEB; the five held-out clusters behind the 22 verdicts
App. D.7, App. C.7pointers from the 22 verdicts to their breakdown and to the near-miss controls
App. B.4, App. E.5re-diagnosis enabled in every reported run; what its assignment pass reads; support by issues
Table E.4 (panel a), Table E.6verdicts against same-fix labels and against repair transfer
AuthorsOpenReview ↗

[W2] "Two Timeouts May Need Different Fixes" versus Merge-Only Clustering [part 1/1]

Thank you, you are right: the sentence overstated what the clustering stage does. Our direct answer: A seed group requires an identical signature (Eq. B.7), and the merge pass can only join groups (Eq. B.8), so the clustering stage cannot separate two issues that share a symptom but need different fixes. The only component that can is the validation-triggered re-diagnosis, which the submission mentioned as "optional" and never evaluated. We corrected the text in §2, §3.3 and App. B.3, and now quantify the problem and how much of it re-diagnosis recovers.

1. How large is the problem? In the blinded same-fix annotation (App. E.4), 157 issue pairs share a signature (98 on ALFWorld, 59 on WebShop). The annotators placed 21 of them (13.4%; 10 and 11) in different fix groups. These are your "two timeouts": same category, span kind, tool, and error token, different fix. Clustering keeps all of them together. The merge pass adds its own errors: 11 of its 160 merge decisions on ALFWorld and 3 of 80 on WebShop join seed groups of different fix groups.

2. What can separate them: validation, then re-diagnosis (App. B.4, App. E.5). When the representative's hypothesis is replayed on other members and a verdict is not confirmed, the cluster becomes mixed or refuted and is regrouped by observed mechanism. Re-diagnosis is behind a configuration switch, hence "optional"; App. B.4 now states that it is enabled in every reported run.

ALFWorldWebShop
Multi-trace clusters3918
Mixed or refuted at the first pass225
Re-diagnosis: revised / split / unassigned only12 / 7 / 33 / 2 / 0
New mechanism groups92
B-cubed precision, before and after0.872 and 0.9670.832 and 0.877
B-cubed precision, multi-trace clusters only0.824 and 0.9540.798 and 0.851
B-cubed recall, before and after0.656 and 0.6390.799 and 0.789
Pairwise precision, before and after0.73 and 0.940.77 and 0.80

Scores are against the blinded same-fix partition (Table E.3).

How a few checked members yield a partition of all issues (App. E.5). Re-diagnosis proposes mechanism groups from the recorded verdicts; an assignment pass then reads every issue of the cluster with its cited span and assigns it to a group or to unassigned, issue by issue. The 22 and 5 re-diagnosed clusters hold 159 and 34 issues in 133 and 31 cluster–trace pairs; the validator had sampled 41 and 13 of these. The pass moved 25 and 5 issues, 13 and 2 of them on never-sampled traces, and moved 25 of 28 and 5 of 5 issues whose reference fix group differs from the representative's. All other clusters keep every issue.

Executed repairs agree: of 17 members a cluster's first fix did not resolve, re-diagnosis placed 12 in 5 second-mechanism groups, and one further patch per group transfers to 9 of the 12 (75.0%; App. E.6).

3. What this does not solve.

  • Unsampled members. The validator samples at most five members, and a cluster whose sampled members all confirm is never re-diagnosed. A same-signature pair with different fixes inside such a cluster stays merged: by the labels, 114 of the 125 issues on never-sampled members of confirmed clusters share the representative's fix group (91.2%); 11 do not.
  • Single-trace clusters. 101 of the 140 ALFWorld clusters and 27 of the 45 WebShop clusters hold one trace and cannot be validated; they hold 27.2% and 17.2% of the issues.
  • Near-misses. The validator refutes only 64.4% of verified near-miss hypotheses (App. E.2).
  • A confirmed verdict is not a guarantee. 4 of the 67 confirmed members belong to another fix group than the representative (Table E.4, panel a).

4. Why the merge pass stays merge-only. We considered letting the model split seed groups and decided against it (App. B.3): the deterministic groups must survive a model failure, and splitting a signature group requires reading the traces, which only investigation and validation do. The cost is the 13.4% above.

5. The corrected text. §3.3 now reads: "Clustering thus cannot separate issues that share a signature (21 of 157 such pairs, 13.4%, need different fixes in our blinded annotation; Appendix E.4); that is left to the re-diagnosis described next." The L131 sentence stays as a statement of the target, now followed by which stage can act on it.

Summary of manuscript changes for W2 (blue in the revised PDF):

LocationChange
§2, §3.3, App. B.3clustering can only join issues; share of identical-signature pairs that need different fixes; reasons for the merge-only constraint
App. B.4, App. E.5, Table E.4 (panel b)re-diagnosis enabled in all runs; how often it revises, splits, or leaves issues unassigned; what its assignment pass reads
Table E.3, §6partition quality before and after re-diagnosis; the merge-only limit listed in §6
Reviewer_JCX9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes TraceACE, which analyzes failures across multiple agent traces, groups issues that may share a fix, and checks proposed explanations against trace evidence and other runs. It reports stronger failure-detection coverage on TRAIL, but mixed results on AgentErrorBench. The paper does not test whether applying the same repair to failures in a group actually resolves them, and its many components make the contribution hard to follow.

Critical strengths
  • TraceACE tests with different backbone LLMs, and it removes individual components to see which ones help. The evaluation also checks whether TraceACE's explanation matches the cause recorded in the benchmark, not just whether its quotations are accurate.

  • It shows the log entries behind its diagnoses and checks that the quoted text actually appears there. It also records why suspected errors were removed, making the reports easier for developers to check.

Critical weaknesses
  • The paper is hard to follow with unclear symbols and formulations, and images (I don't see the purpose of the image under figure 1), and its advance over prior work is unclear. For example, Agentic CLEAR already analyzes recurring issues across traces, and CORRECT reuses recurring error patterns. TraceACE investigates a representative trace and checks its explanation against other traces, but the paper does not clearly establish the additional benefit of this combination.

  • The paper groups failures that may share a fix, but does not verify this by applying a proposed repair. Showing that an explanation fits several logs does not establish that one repair would resolve those failures.

  • Section 3.4 requires every quotation to be verifiable before accepting an explanation, but Table 3 still reports grounding failures.

Questions for authors

Please see the weaknesses above.

10 comments

Given the lack of further discussion with the authors, my original concerns remain unresolved. Therefore, I will keep my score unchanged.

AuthorsOpenReview ↗

[W1] Presentation, the Image Under Figure 1, and the Advance over Agentic CLEAR and CORRECT [part 1/1]

Thank you, the presentation problems you list are real. Our direct answer: We kept the image under Figure 1 and gave it the purpose it lacked: it is now Figure 2, a worked example that follows one case from input to output; we added a notation table and defined what the text left unspecified (gate failure and the per-trace readout); and we now state and measure the advance over Agentic CLEAR and CORRECT. Both group recurring issues by resemblance. TraceACE groups by shared fix and tests each cluster's explanation on the cluster's other traces: on the same issue records, CLEAR-style synthesis reaches a B-cubed F1 of 0.596 and 0.595 against blinded same-fix labels, TraceACE 0.748 and 0.815.

1. The image under Figure 1. It had no number, no caption, and no reference in the text. The image is now Figure 2, referenced where §1 defines the task, and its caption follows one concrete case through its five boxes: "Several traces end in a tool timeout. Analysis locates the failing span in each trace, clustering groups the timeouts into one candidate cluster, explanation tests on the other traces whether they share one triggering mechanism, and the output proposes a fix that still has to be validated." Former Figures 2 and 3 are now Figures 3 and 4.

2. Symbols and formulations.

  • New Table A.1 lists every symbol of §2 and §3 with a short meaning.
  • §3.4 now defines what was left open: a hypothesis that fails the gate after two retries is kept but marked unadmitted (reply to W3), and a fixed precedence selects one hypothesis per trace when a benchmark needs one.
  • §2 and §3.3 say what clustering cannot do: it only joins issues; separating issues that share a signature is left to re-diagnosis.

3. Advance over Agentic CLEAR and CORRECT (§1; new Table G.1, three columns shown):

Agentic CLEARCORRECTTraceACE
What is carried across tracesissues clustered and summarized from step- and trace-level critiqueserror schemata distilled from clusters of earlier failures and cachedclusters of issues that one fix should address
Evidence behind a diagnosistextual critiques, each issue linked to the step or trace that triggered itcached schemata retrieved by similarityspan identifiers and quotations checked by a gate
How a diagnosis is testedno test step is describedcandidate schemata are replayed on earlier trajectories; the most accurate is kepton the other traces of its cluster: verdicts, status, and re-diagnosis
Same-fix grouping against blinded labels (B-cubed F1, ALFWorld / WebShop)0.596 / 0.595—0.748 / 0.815
Other members of the group that the same ten patches reach38.2%—75.3%

The first three rows follow the cited papers; the last two are measured on our adaptations (App. E.4, Table E.8). CORRECT's cache is frozen (Table C.3), so its replay step was not run. Per patch, CLEAR-style groups resolve almost as many (5.8 against 6.4): the advance is more precise coverage within one workflow, not a new clustering mechanism.

4. Two further measured differences (App. G.5, App. E):

  • Testing an explanation across traces carries information. Members the validator confirms share the representative's fix group in 94.0% of cases, refuted ones in 36.4%. When a cluster's fix is executed, it transfers to 91.7% of the members it was not written from in confirmed clusters and 20.0% in refuted ones (Tables E.4, E.6).
  • End to end. Ten patches read off TraceACE's cluster reports raise the agent's success by 7.7 points over four task sets, two of them held out; ten read off per-trace reports, by 3.3. A lead over patches for CLEAR-style groups is not established when an author writes them (+1.8). It is when a patcher generates them from each workflow's own output (+2.9 [+0.8, +4.9]), but it falls to +1.4 [−0.7, +3.5] once both get an investigated report: it belongs to the investigation, not the grouping (Table E.9).

5. Where it does not help. Root accuracy on AgentErrorBench is not better than per-trace investigation (pooled: +0.16 [−2.11, +2.44], inside a preset ±3\pm3-point margin) at 3.1 and 2.4 times the tokens, and under GPT-5.5 Agentic CLEAR remains ahead on TRAIL SWE (pair F1 23.7 vs. 14.8). The validator refutes only 64.4% of verified near-miss hypotheses (App. E.2), and a budget advantage over simpler grouping pipelines is not established (App. E.6).

Summary of manuscript changes for W1 (blue in the revised PDF):

LocationChange
§1, Figure 2image under Figure 1 numbered, captioned with a worked example, and referenced in §1
App. A (Table A.1), §2, §3.3, §3.4notation table; what clustering can and cannot separate; gate failure; per-trace readout
§1, §5, App. G.5, Tables G.1, E.8, E.9positioning against the four closest systems, with measured differences
AuthorsOpenReview ↗

[W2] Verifying a Group by Applying Its Repair [part 1/5]

Thank you, we agree that an explanation that fits several logs does not show that one repair would resolve those failures, so we applied repairs. Our direct answer: We cannot re-run the agents behind TRAIL and AEB with a change, so we built a re-runnable cohort. For ten clusters we implemented the cluster's fix and re-ran every member task three times. The fix removes the target issue in 76.5% of re-runs (16.5% with no change, 20.7% with a mismatched fix) and transfers to 75.3% of the members it was not written from. It does not transfer to all, and how well it transfers is predicted by the validator's status. Part 2/5 shows that this does not rest on our analyzer's judgment or on ten clusters only; parts 3/5 to 5/5 cover the patched agent, other groupings and attribution.

1. Setup (App. E.6). A ReAct-style agent (public ReAct prompts, DeepSeek V4 Flash) was run on the 134 unseen ALFWorld games (85 solved) and the first 200 WebShop test instructions (reward 1 on 68). TraceACE analyzed the 181 failed episodes. The selection rule was fixed before any re-run: every cluster with at least five member traces whose admitted hypothesis proposes a scaffold-level fix. Thirteen clusters have that size; three were excluded because their fix needs another model or a changed environment. For each of the ten remaining clusters one author implemented the report's fix sketch as a single patch, having seen the representative trace but none of the other members. All 95 cluster–member pairs were re-run three times per arm: no change, the cluster's own fix, and the patch of another cluster.

2. Per cluster (Table E.5; m is the number of member traces; Transfer counts members whose target issue is absent in at least two of three re-runs with the cluster's own fix, Others the same without the representative, from which the fix was written; the last three columns count re-runs without the target issue, out of 3m):

MechanismmStatusTransferOthersOwn fixNo changeMismatched
A1repeats an inadmissible action9confirmed8/97/824/273/274/27
A2alternates between two receptacles7mixed5/74/614/215/215/21
A3uses an appliance before holding the object6confirmed6/65/517/182/183/18
A4stops searching after the first location5mixed3/52/49/153/154/15
W1buys without selecting a required option16confirmed15/1614/1544/486/488/48
W2search query copies the full instruction14mixed10/149/1329/428/429/42
W3ignores the price ceiling12confirmed11/1210/1132/365/367/36
W4selects from the first result page only10mixed7/106/921/305/307/30
W5ends the episode on a mismatched product9refuted3/92/810/276/277/27
W6clicks an element that is not on the page7confirmed6/75/618/214/215/21
All9574/9564/85218/28547/28559/285

A1–A4 are ALFWorld and W1–W6 WebShop clusters. The patches are small scaffold changes, e.g. an admissible-action check in the action wrapper (A1) or a price guard before purchase (W3).

3. What the re-runs show (Table E.6).

  • One repair resolves most members of a group. The own fix removes the target issue in 218 of 285 re-runs (76.5%), against 47 (16.5%) with no change and 59 (20.7%) with another cluster's patch.
  • It carries over to traces it was not written from. Each fix was written from the representative, so transfer is counted over the other members: 64 of 85 (75.3%), against 9 (10.6%) with no change and 12 (14.1%) with a mismatched fix. With the ten representatives the rate is 74 of 95 (77.9%).
  • Not all groups are equal, and validation tells them apart. Among the other members the fix transfers to 41 of 45 (91.1%) in the five confirmed clusters, 21 of 32 (65.6%) in the four mixed ones, and 2 of 8 (25.0%) in the refuted one.
  • Not one big cluster. The own fix beats no change in each of the ten clusters. Per cluster, the rate among the other members ranges from 25.0% to 100.0%, and resampling the ten clusters gives a 95% interval of [60.2, 86.6] for the 75.3%.
  • Removing the issue is not the same as solving the task. Task success rises to 115 of 285 re-runs (40.4%) from 28 (9.8%); failed episodes usually carry other issues.

Summary of manuscript changes for W2, part 1 (blue in the revised PDF):

LocationChange
§4.6, Table 4executed repairs; transfer reported for members a fix was not written from
App. E.6, Tables E.5 and E.6, Figure E.2cohort, selection, patches, arms; transfer per cluster and by status, with and without representatives
Abstract, §6, App. G.6"interventional validation remains unavailable" replaced by the result and its limits
Reviewer_FH6EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TraceACE combines specialized error scanners, interpretable “same-fix” clustering, representative-trace investigation, quotation verification, and cross-trace hypothesis checking. It reports substantial improvements on TRAIL’s joint detection metric across multiple backbones, with more mixed results on critical root-cause attribution.

Critical strengths

Preserving source span identifiers and mechanically checking quotations are useful safeguards against ungrounded diagnoses. The evaluation spans eight backbones and seven baselines, and the ablations help distinguish detection from explanation components. The authors also explicitly distinguish evidence grounding from causal correctness.

Critical weaknesses

The headline detection metric is incomplete. Appendix C defines “Joint” as recall over reference location–category pairs, not precision-sensitive accuracy. Extra predicted locations are unpenalized, and category-level F1 does not penalize extra locations within an otherwise correct category. Consequently, the reported gains do not establish improved issue-level precision. Pair-level precision/F1 and predictions-per-trace are important missing diagnostics.

The central same-fix claim lacks direct validation. No developer-confirmed same-fix labels or executed repair interventions establish that clustered issues genuinely share an effective correction. Recoverable quotations and cross-trace agreement support observational consistency, not repair validity. The negative controls reject clearly irrelevant hypotheses rather than plausible competing mechanisms.

Collection-level benefits are small relative to their cost. In Table 3, improvements over per-trace investigation correspond to only one additional correctly attributed typed trace in each environment. Post-analysis token costs increase roughly threefold on ALFWorld and more than twofold on WebShop; ALFWorld also produces more primary reports. These single-run estimates need uncertainty analysis and budget-matched comparisons.

Questions for authors

Does the TRAIL advantage survive precision-sensitive pair-level scoring? Can a blinded same-fix annotation or executed-repair study validate the clustering contribution?

10 comments
AuthorsOpenReview ↗

[W1 & Q1] Pair-Level Precision and F1: Does the TRAIL Advantage Survive? [part 1/1]

Thank you, you are right about the metric. Our direct answer: Joint is recall over reference pairs, and TraceACE does predict more pairs than the reference contains on GAIA (6.3–7.3 against 4.9 per trace). Under pair-level F1 its GAIA lead survives under all eight backbones but shrinks from 24.2 Joint points to 12.2 F1 points; on SWE it leads under seven backbones, two of them by at most 1.4 points. Pair-level precision, F1 and predictions per trace are now reported for all 64 method–backbone cells (new Table D.7).

1. What we computed (App. C.1, Eq. C.5; App. D.8). No new model call was needed: we re-scored the prediction files behind Table 1. For a trace with reference pairs GG and predicted pairs G^\hat G, pair precision is ∣G∩G^∣/∣G^∣|G\cap\hat G|/|\hat G| (zero for an empty prediction set), pair recall is ∣G∩G^∣/∣G∣|G\cap\hat G|/|G| (this is Joint), and pair F1 is 2∣G∩G^∣/(∣G∣+∣G^∣)2|G\cap\hat G|/(|G|+|\hat G|). All three are macro-averaged over traces exactly as for Joint, so F1 is averaged per trace and is not the harmonic mean of the P and R columns below. We add the mean number of predicted pairs per trace.

2. The DeepSeek V4 Flash block of Table D.7 (GAIA: 116 traces, 4.9 reference pairs per trace; SWE: 31 traces, 8.3 pairs per trace; R is the Joint column of Table 1):

MethodGAIA Pred.PRF1SWE Pred.PRF1
LLM3.130.419.621.22.613.24.36.1
ReAct4.232.928.527.93.013.05.16.9
CORRECT2.625.714.417.01.11.60.10.2
AgentDebug5.224.725.722.92.28.43.04.2
AgentRx3.829.621.922.35.416.711.012.2
AgentDebugX1.039.77.512.41.09.71.22.1
Agentic CLEAR4.420.218.516.911.719.831.622.6
TraceACE7.334.650.939.112.223.836.126.5

3. Across the eight backbones.

  • Predictions per trace. TraceACE predicts 6.3–7.3 pairs per GAIA trace and 7.1–12.5 per SWE trace. The baselines predict 0.8–5.5 and 0.4–12.2, and AgentDebugX exactly one, because it keeps its root-only prediction. Part of the Joint lead on GAIA is therefore bought with extra predictions, which is the effect you anticipated.
  • Precision. TraceACE's pair precision on GAIA is 30.2–38.1%. That is comparable to the best baselines but not the highest: AgentDebugX's single prediction is more precise under six backbones, ReAct under three, and LLM under two.
  • F1 on GAIA. TraceACE has the highest pair F1 under all eight backbones (32.2–42.2). Its margin over the best baseline by F1 averages 12.2 points (range 7.8–17.4), about half of the 24.2-point Joint margin (range 18.4–30.1).
  • F1 on SWE. TraceACE is highest under seven of eight backbones; averaged over all eight, the margin is 4.0 points (Joint margin 5.6). Two of these leads are small (+0.1 under Claude Opus 4.6 and +1.4 under Gemini 3.7 Flash), and under GPT-5.5 Agentic CLEAR stays ahead by 8.9 points (23.7 vs. 14.8). We now describe SWE as a lead under five backbones, a negligible lead under two, and a deficit under one.

Pair F1 of TraceACE against the strongest baseline by F1, per backbone (Table D.7):

BackboneGAIASWE
GPT-5.542.2 vs. 24.814.8 vs. 23.7
Claude Opus 4.641.3 vs. 29.423.0 vs. 22.9
Gemini 3.7 Flash32.2 vs. 20.920.0 vs. 18.6
DeepSeek V4 Flash39.1 vs. 27.926.5 vs. 22.6
Qwen 3.8 Flash38.9 vs. 23.327.7 vs. 22.5
Kimi K340.3 vs. 29.530.1 vs. 22.2
Doubao Seed 2.0 Pro35.6 vs. 24.131.9 vs. 15.1
GLM 5.3 Flash36.9 vs. 29.126.4 vs. 20.8

4. What changed in the claims. The abstract and §4.2 now state the F1 result next to the Joint result, and §4.1 says explicitly that Loc. and Joint are recall over reference items and do not penalize extra predictions. Joint stays in Table 1 as the benchmark's metric, now with its precision-sensitive companion. App. C.6 now also states your second observation: category-level F1 does not penalize extra locations inside a correct category.

5. Limits. Pair precision treats every predicted pair that is not annotated as wrong. If the reference annotation of a trace is incomplete, precision is a lower bound for every method; we did not estimate annotation completeness. Each cell of Table 1 is still one run per method and backbone.

Summary of manuscript changes for W1 & Q1 (blue in the revised PDF):

LocationChange
Abstract, §4.1, §4.2Joint identified as recall; pair-level F1 lead stated with its reduced margins (GAIA 12.2, SWE 4.0)
App. C.1 (Eq. C.5), App. C.6pair precision and F1 defined; limit of category-level F1 stated
App. D.8, Table D.7, Figure D.4pair-level P, R, F1 and predictions per trace for 8 methods ×\times 8 backbones ×\times 2 cohorts
§6conclusion reworded: a smaller but persistent lead in pair-level F1
AuthorsOpenReview ↗

[W2 & Q2] Direct Validation of the Same-Fix Claim: Annotation, Executed Repairs, and Controls [part 1/6]

Thank you, the submission offered no direct evidence for the same-fix claim, and we ran both studies you ask for. Our direct answer: (i) In a blinded, adjudicated same-fix annotation of all 534 issues of the two AEB runs, TraceACE's clusters reach a B-cubed F1 of 0.748 (ALFWorld) and 0.815 (WebShop), the best of six grouping methods before re-diagnosis; precision is 0.872 and 0.832, and recall is the weaker side (0.656 and 0.799). (ii) Executed repairs follow in parts 2 to 5; (iii) near-miss controls and the scope of our claims in part 6/6.

1. Protocol (App. E.4). Two annotators with agent-engineering experience had no access to signatures, clusters, hypotheses, or validator verdicts. They received every issue that survived post-processing in the DeepSeek V4 Flash runs: 371 ALFWorld issues from 98 traces and 163 WebShop issues from 49 traces. Each issue was a card (category, cited span, summary, task) in random order. The instruction was to sort the cards of a cohort into groups such that one concrete change to the agent (a prompt rule, a wrapper or parser fix, a guard, or a policy change) would address every issue in the group.

  • Agreement and reference. Cohen's κ\kappa over all issue pairs is 0.81 on ALFWorld and 0.74 on WebShop (B-cubed F1 between the two annotators' partitions: 0.81 and 0.77). A third annotator adjudicated the disagreements, giving 58 and 27 reference fix groups.

2. Clustering accuracy against the adjudicated partition (Table E.3; every row partitions the same issues; K=number of clustersK=\text{number of clusters}):

GroupingALFWorld KPRF1WebShop KPRF1
Category only100.2990.8910.44780.3570.8410.501
Signature only3000.9750.1900.3181250.9570.2210.359
TF-IDF agglomerative530.5680.4790.520270.6480.6420.645
LLM-only, one pass440.6010.5660.583220.6240.6700.646
CLEAR-style issue synthesis350.5320.6780.596120.5000.7330.595
TraceACE1400.8720.6560.748450.8320.7990.815
TraceACE + re-diagnosis1540.9670.6390.770490.8770.7890.830

In the last row the 5 and 2 issues that re-diagnosis leaves unassigned are scored as singletons, so KK is 140+9 new groups+5=154140+9\text{ new groups}+5=154 and 45+2+2=4945+2+2=49. Re-diagnosis reads every issue of a re-diagnosed cluster, not only the sampled members (App. E.5).

3. How to read it.

  • Precision, and what it does not mean. Of TraceACE's 160 and 80 merge decisions, 149 and 77 join seed groups of one reference fix group. B-cubed precision is not the reliability of a multi-trace cluster: 101 and 27 clusters hold a single trace, and all but one of them a single issue, which is pure by definition. Over the 270 and 135 issues in multi-trace clusters it is 0.824 and 0.798 (0.954 and 0.851 after re-diagnosis); pairwise precision is 0.73 and 0.77.
  • Recall is the weaker side. 140 and 45 clusters stand against 58 and 27 reference groups: the merge pass is conservative and leaves fix groups split.
  • A shared signature is not a shared fix. Of the 157 issue pairs with identical signatures (98 on ALFWorld, 59 on WebShop), the annotators placed 21 (13.4%; 10 and 11) in different fix groups.
  • Alternatives group by topic. TF-IDF, one-pass LLM, and CLEAR-style grouping reach at most 0.648 precision; of the three, only CLEAR-style synthesis on ALFWorld exceeds TraceACE's recall (0.678 against 0.656).

4. Do the validator's verdicts track the labels? For each of the 99 secondary-trace verdicts we looked up whether the member's issue shares the reference fix group of the representative's issue (Table E.4, panel a):

VerdictSame fix groupDifferentSame fix
Confirmed63494.0%
Partial14766.7%
Refuted4736.4%

Cross-trace agreement is therefore informative about the same-fix relation, not only about observational consistency. It is not proof: 4 of 67 confirmed members belong to another fix group.

5. Limits. One backbone and two environments. The annotators judge whether one change would fix a group without executing it (parts 2 to 5 execute the repairs). The reference partition is itself a judgment (κ\kappa of 0.74 on WebShop); differences of a few hundredths between rows should not be over-read.

Summary of manuscript changes for W2 & Q2, part 1 (blue in the revised PDF):

LocationChange
§4.6 (new), Table 4annotation protocol, B-cubed scores, verdicts against labels
App. E.4, Tables E.3 and E.4, Figure E.1protocol, seven groupings, pairwise scores, precision without single-trace clusters, verdict-by-label counts
§3.3, App. B.3clustering cannot split a signature group; 13.4% of identical-signature pairs need different fixes
AuthorsOpenReview ↗

[W2 & Q2] Direct Validation of the Same-Fix Claim: Annotation, Executed Repairs, and Controls [part 2/6]

Our direct answer on executed repairs: On a re-runnable cohort, a cluster's own fix removes the target issue in 76.5% of member re-runs, against 16.5% with no change and 20.7% with a mismatched fix. It transfers to 64 of the 85 members it was not written from (75.3%), and the validation status predicts to which (91.1% in confirmed clusters, 25.0% in the refuted one). This does not rest on our analyzer's signature: it holds at the category level, in task success, and in a blinded audit of the mechanism.

1. Design (App. E.6). The agents behind TRAIL and AEB cannot be re-run with a change, so we built a new cohort.

  • Cohort. A ReAct-style agent (public ReAct prompts, DeepSeek V4 Flash) was run on the 134 unseen ALFWorld games (85 solved) and the first 200 WebShop test instructions (reward 1 on 68). TraceACE analyzed the 181 failed episodes (31 and 58 clusters).
  • Selection, fixed before any re-run. Every cluster with at least five member traces whose admitted hypothesis proposes a scaffold-level fix. Thirteen clusters have that size; three were excluded because their fix needs another model or a changed environment. Ten clusters and 95 cluster–member pairs remain; part 3/6 adds 13 more.
  • Fix and arms. One author implemented each fix sketch as a single patch (Table E.5), having seen the representative but no other member. Every pair was re-run three times per arm: no change, the cluster's own fix, and another cluster's patch (285 re-runs per arm).
  • Units. The 95 pairs involve 81 distinct tasks (23 on ALFWorld, 58 on WebShop); 14 tasks lie in two clusters. The 285 re-runs per arm share ten patches and are not independent, so point 3 reports results per cluster.

2. Is the issue really gone? At the signature level, "absent" means that the frozen Failure Analyzer finds no issue with the cluster's signatures. A patch can change the form of an error without removing its mechanism, so we add readings independent of the signature (Table E.6, panel a):

ArmAbsent, signature levelAbsent, category levelTask successTransfer, other members
No change47/285 (16.5%)41/285 (14.4%)28/285 (9.8%)9/85 (10.6%)
Mismatched fix59/285 (20.7%)52/285 (18.2%)33/285 (11.6%)12/85 (14.1%)
Cluster's own fix218/285 (76.5%)202/285 (70.9%)115/285 (40.4%)64/85 (75.3%)
  • Category level: no issue of the cluster's taxonomy category anywhere in the re-run (conservative: an unrelated issue of that category counts as present).
  • Task success is the environment's own criterion, with no component of TraceACE. It stays at 40.4% since failed episodes usually carry other issues.
  • Blinded audit (new Table E.7). An author who had not written the patches audited 60 re-runs, 20 drawn at random per arm. The auditor saw the trace and the mechanism of the cluster report, but neither the arm nor the analyzer's verdict, and judged whether that mechanism still occurs in any form, not whether the signature matches:
ArmAnalyzer: absentAuditor: absentAgreeAnalyzer absent, auditor presentAnalyzer present, auditor absent
No change3/202/201910
Mismatched fix4/204/201811
Cluster's own fix15/2014/201910

They agree on 56 of 60 re-runs (93.3%); the four disagreements are spread over the arms. In the own-fix arm it is a displacement: the mechanism reappeared in another form after the patch.

3. Transfer across traces (Table E.6, panel b). A fix transfers to a member when the issue is absent in at least two of three re-runs. Each fix was written from the representative, so we exclude it; with the ten representatives the rate is 74 of 95 (77.9%).

Members other than the representativeMembersTransferred
Confirmed clusters (5)4541 (91.1%)
Mixed clusters (4)3221 (65.6%)
Refuted cluster (1)82 (25.0%)
Confirmed / partial / refuted verdict20 / 7 / 719 (95.0%) / 4 (57.1%) / 2 (28.6%)
Never sampled by the validator5139 (76.5%)

Per cluster the rate ranges from 25.0% (the refuted cluster) to 100.0%, and the own fix beats no change in each of the ten clusters. Resampling clusters instead of re-runs gives a 95% interval of [60.2, 86.6] for the 75.3% and of [48.2, 77.7] for its difference to no change (64.7 points).

Summary of manuscript changes for W2 & Q2, part 2 (blue in the revised PDF):

LocationChange
§4.6, Table 4executed repairs; transfer reported for the members a fix was not written from
App. E.6, Tables E.5–E.7, Figure E.2cohort, selection, patches, arms, three readings, blinded audit, transfer with and without representatives
Abstract, §6"interventional validation remains unavailable" replaced by the result and its limits

Comments on the paper

2 comments
AuthorsOpenReview ↗

[General Response] What the Revision Adds, What It Concedes, and Where Each Concern Is Answered [part 1/2]

We thank the three reviewers, their criticisms were specific and, in the main, correct. Our direct answer: We ran the experiments the reviews ask for, withdrew or narrowed four claims that the new evidence does not support, and rewrote what was hard to follow. No previously submitted number was changed. This part lists the new evidence by concern; part 2/2 states which claims changed, what is established and what is not, and the limits.

New evidence, by concern (DeepSeek V4 Flash unless noted):

ConcernRaised inWhat we didResultWhere
Joint is recall; extra predictions are not penalizedFH6E W1pair-level precision, F1 and predictions per trace for all 64 cellsGAIA F1 lead under all eight backbones, 12.2 points on average (Joint: 24.2); SWE lead under seven (4.0 averaged over all eight)Table D.7
No same-fix labels, no collection-level metricFH6E W2, YUXj W1blinded, adjudicated annotation of 534 issues (κ\kappa 0.81 and 0.74)B-cubed F1 0.748 and 0.815, the best of six groupings; recall is the weaker sideTable E.3
No executed repairFH6E W2, JCX9 W223 repaired clusters, three arms, three re-runs, a blinded auditin ten clusters the own fix removes the issue in 76.5% of re-runs (16.5% with no change) and transfers to 75.3% of the members it was not written from; over all 23 clusters to 73.5% (91.7%, 66.7%, 20.0% in confirmed, mixed, refuted ones)Tables E.5–E.7
Does repair help the agent?FH6E W2, JCX9 W2ten patches applied together; analyzed and held-out task sets; author-written and automatically generated patchessuccess +7.7 points over four task sets [+5.5, +10.0]; +3.3 with ten patches from per-trace reports; a lead over two other groupings only for generated patches (+2.9, +2.8)Table E.9
Is that lead due to the clustering?all reviewsgrouping crossed with what the patcher receives; patch sets regenerated twiceno: with an investigated report for every grouping the lead is +1.4 and +1.3, not established; a report instead of a description adds +1.7 [+0.6, +2.9]Table E.9, panels b, c
Cost against gainFH6E W3tokens counted on the cohort where the gains were measured1.4 times per-trace RCA for more than twice the gain; more than ten times a grouping pipeline with ten investigated representatives, whose gain is not reliably lowerTable E.9, panel b
Would another grouping be repaired as well?JCX9 W1, YUXj W3fixed patches applied across groupings in both directions75.3% of a cluster reached against 38.2% and 45.5% of the other groups (per patch 6.4 against 5.8 and 5.5); their own patches reach 68.4% and 69.1% of a TraceACE clusterTables G.1, E.8
Controls reject only obvious negativesFH6E W2near-miss hypotheses on 25 clusters, verified as negativesof 73 verified negatives 6.8% confirmed and 64.4% refuted; real hypotheses on the same traces: 72.0% confirmedTable E.2
Single runs, small gainsFH6E W3five runs, two more arms, a preset equivalence margin of ±3\pm3 pointsagainst per-trace RCA +0.16 [−2.11, +2.44], inside the margin; against analysis-only +5.04 [+2.11, +8.62]Table D.8
Grounding below 100 despite the gateJCX9 W3gate outcome of every report9.3% and 11.1% of reports stay unadmitted and hold every ungrounded itemTable E.11
13 of 22 verdicts; merge-only clusteringYUXj W1, W2cluster status, re-diagnosis outcomes, identical-signature pairs27 of 57 multi-trace clusters are not confirmed at the first pass; re-diagnosis raises precision to 0.967 and 0.877; 13.4% of identical-signature pairs need different fixesTables E.3, E.4
Where AEB answers come fromYUXj Q1provenance of every answer11 of 100 and 9 of 50 answers come without investigation or validation of the trace; none of them is All-correctTable E.10
PresentationJCX9 W1worked example for the image under Figure 1 (now Figure 2); notation table; gate failure and per-trace readout definedTable A.1, §3.4

How to read the numbers. The repair study uses a re-runnable cohort of a ReAct-style agent (134 ALFWorld games, 200 WebShop instructions; 181 failed episodes), because the agents behind TRAIL and AEB cannot be re-run with a change. Transfer rates are conditional on the selection rule (clusters with at least three member traces whose hypothesis yields a scaffold-level patch; seven clusters were excluded) and are not repair rates over all failed tasks; the 23 repaired clusters cover 112 of the 181 failed episodes (61.9%). The agent's success is measured separately on four task sets, two of them held out. All new studies use one analysis backbone and one agent scaffold; the second agent backbone runs on the same scaffold with the same ten patches. Intervals are 95% bootstrap intervals over the unit named with each result.

AuthorsOpenReview ↗

[General Response] What the Revision Adds, What It Concedes, and Where Each Concern Is Answered [part 2/2]

Our direct answer on what the paper claims after the revision: The contribution is an integrated workflow whose clusters agree with same-fix labels, whose validation status predicts whether a fix transfers, and whose clusters state more precisely than nearby groupings which traces a patch resolves. It is not a better clustering mechanism, it does not raise root accuracy, and it is not shown to be budget-efficient against simpler grouping pipelines.

1. Claims we changed.

  • Root accuracy. Cluster-level investigation does not raise it: over five runs it is equivalent, pooled, to per-trace investigation within ±3\pm3 points, at 3.1 and 2.4 times the post-analysis tokens.
  • The TRAIL lead. It is now stated with its pair-level F1 margins: about half the Joint margin on GAIA; on SWE a lead under five backbones, a negligible lead under two, a deficit under one.
  • What clustering can do. Signatures and the merge pass can only join issues; 13.4% of identical-signature pairs need different fixes, and only re-diagnosis can separate them.
  • What the tokens buy. We no longer say that the extra tokens buy the repair gains. The end-to-end lead over nearby groupings comes mainly from the investigation report, and a budget advantage is not established.

2. What is established, and what is not (repair cohort unless noted; details in the point-by-point replies):

ClaimEvidenceStatus
Clusters agree with blinded same-fix labels better than five other groupingsB-cubed F1 0.748 and 0.815 against at most 0.646 (AEB)established on 534 issues
A cluster states more precisely which traces a patch resolvesfixed patches in both directions: 75.3% against 38.2% and 45.5%; 68.4% and 69.1% against 39.9% and 45.6%; generated patches at equal input: 63.5% against 34.1% and 39.5%established
The validation status predicts whether a fix transfers91.7%, 66.7%, 20.0% in confirmed, mixed, refuted clusters; r=0.89r=0.89established on 23 clusters
Fixes read off cluster reports help the agent, also on held-out tasks+7.7 [+5.5, +10.0] with author-written, +5.8 [+3.6, +8.0] with generated patchesestablished on one scaffold
Choosing fixes by cluster beats choosing by single trace+4.4 [+2.2, +6.6] and +3.4 [+1.2, +5.5]established
The whole workflow beats nearby grouping workflows end to endauthor-written: +1.8 [−0.4, +3.9] and +2.1 [0.0, +4.3]; generated: +2.8 [+1.1, +4.5] and +2.7 [+1.0, +4.4] over three generationsonly with generated patches
The clustering itself beats nearby groupings end to endequal patcher inputs: +1.4 [−0.7, +3.5] and +1.3 [−0.8, +3.4]not established
More repair per patch6.4 against 5.8 and 5.5 other membersnot established
Higher root accuracy than per-trace investigation+0.16 [−2.11, +2.44] (AEB)withdrawn (equivalent, pooled)
Budget efficiency against simpler grouping pipelines169.3M tokens against 13.7M and 11.4M, for gains of 5.8 against 4.4 and 4.5 pointsnot established
Signature, merge, validation, and re-diagnosis are each necessary for the repair gainsno component was removed in the repair studynot tested

3. Limits we state.

  • Scope. One analysis backbone and one agent scaffold; the second agent backbone runs on the same scaffold with the same patches.
  • Selection. 23 repaired clusters, conditional on the selection rule (seven clusters were excluded because their fix is not a scaffold-level patch, and clusters with one or two traces were not repaired); a transfer rate is not the chance that an arbitrary failure is repaired, and after ten patches most runs of the previously failed tasks still fail.
  • Attribution. A patch author or a patcher stands between every report and its repair, so repair gains belong to the workflow, not to the algorithms alone.
  • Uncertainty. End-to-end intervals resample tasks, and for generated patches also patch sets; each workflow was executed once on the cohort, so they do not cover a repeated diagnosis. The comparisons hold the number of patches fixed, not the total budget.
  • Validation. A confirmed status refers to the sampled members only; 128 of 185 AEB clusters hold one trace and cannot be validated; 27 of the 57 multi-trace clusters are not confirmed at the first pass; verified near-miss hypotheses are refuted only about two thirds of the time.
  • Main table. Table 1 is one run per cell.

4. What we ask. We do not ask the reviewers to credit a new clustering algorithm. We ask whether a workflow that carries a diagnosis from single traces to validated, executable fixes, with its measured benefits and its stated costs, is a useful systems contribution. We hope the added evidence and the narrowed claims resolve the concerns behind the current ratings, and we would be grateful for your reconsideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.