- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses component selection when optimizing LLM agents with editable system prompts, tool descriptions, and experience banks. It formulates this problem as multi-label structural attribution: identifying components that offer feasible intervention points for repairing a failure. CATO contrasts failed and successful executions of the same task under different configurations, produces component scores and textual gradients, and allocates candidate edits according to the accumulated evidence. Each candidate changes one component and is selected using a separate validation set. Experiments cover AppWorld, WebArena, and ten BBH subtasks with three backbones. The paper reports consistent improvements over six optimization baselines, alongside controlled fault-injection experiments and ablations examining the attribution and editing pipeline.
Critical strengths
-
The controlled experiments connect attribution quality to actual repair utility. The fault-injection protocol changes all three components and verifies that the two neutral edits preserve success both individually and jointly. This makes the evaluation more informative than simply identifying the only changed component. CATO achieves 95.7% injected-fault identification accuracy versus 52.8% for trajectory-only diagnosis. Under shared candidate-generation and validation procedures, its repairs also achieve higher failure repair and success retention. This provides substantive evidence that better component selection improves downstream optimization.
-
The end-to-end evaluation supports the practical value of targeted editing. CATO obtains the highest mean score across the reported backbone–metric combinations, including AppWorld applications unseen during optimization. On GLM-5.2, Test-Normal scenario goal completion reaches 89.29%, compared with 83.33% for the strongest competing optimizer. The ablations also show substantial degradation when contrastive attribution, gradient consolidation, or single-component editing is removed. Resource accounting includes rollout and validation costs, demonstrating that the gains are compatible with lower token consumption than several strong baselines.
Critical weaknesses
-
The attribution evaluation covers a narrower setting than the general multi-label formulation. The 108 injected-fault cases come from 12 faulty configurations, with one designated functional fault per configuration and a successful reference execution available. This establishes strong performance at ranking a known injected repair target. It provides less direct evidence about naturally occurring failures with several plausible repair locations or interacting defects. Moreover, the reported accuracy uses the unique highest positive score, whereas optimization routes every component whose score exceeds η = 0.6. Consequently, the headline accuracy does not directly establish the precision and recall of the assignments actually used during optimization.
-
The reliability of the smaller comparative gains is insufficiently quantified. Table 1 reports means over three independent runs without corresponding dispersion or confidence intervals. This matters particularly for the BBH improvements of approximately 0.76–1.17 percentage points and the smaller WebArena margins. The larger AppWorld improvements are more compelling, but the claim of consistent superiority would be stronger with uncertainty estimates for paired differences. Figure 6’s variation across population members does not substitute for variability across independent optimization runs.
Questions for authors
-
Under the actual assignment rule of scores greater than 0.6, what are the precision and recall for identifying the injected component, and how often are additional components assigned? Can you provide uncertainty estimates that account for cases sharing the same faulty configuration?
-
During ordinary optimization, what fraction of attributed failures have a successful same-task reference, and how does that fraction change across rounds and benchmarks? How much of the improvement comes from contrastive attribution versus the failure-only fallback?
-
Please provide per-run results or confidence intervals for the differences against the strongest baselines, especially on WebArena and BBH. Are the smaller gains consistent across all three independent runs?
2 comments
Response to Reviewer BUUo (Part 2/2)
Q2: During ordinary optimization, what fraction of attributed failures have a successful same-task reference across rounds and benchmarks, and how much of the improvement comes from contrastive attribution versus the failure-only fallback?
A2: On GLM-5.2, 40.9–52.3% of analyzed failures have a successful same-task reference (contrast pairs) across all 6 rounds, and 60.0–68.2% in Rounds 2–6. Round 1 starts from a single seed (), so no reference exists and its share is 0%. Entries are contrast pairs / analyzed failures (Table 10):
| Benchmark (GLM-5.2) | Round 1 | Rounds 2–3 | Rounds 4–6 | Rounds 2–6 | All rounds |
|---|---|---|---|---|---|
| AppWorld | 0/7 | 7/9 (77.8%) | 2/6 (33.3%) | 9/15 (60.0%) | 9/22 (40.9%) |
| WebArena | 0/10 | 15/19 (78.9%) | 7/14 (50.0%) | 22/33 (66.7%) | 22/43 (51.2%) |
| BBH (10 subtasks) | 0/52 | 76/97 (78.4%) | 40/73 (54.8%) | 116/170 (68.2%) | 116/222 (52.3%) |
The share peaks in Rounds 2–3, when most population updates occur, and tapers in Rounds 4–6 once configurations have largely converged and the remaining failures are mostly shared hard tasks. From Round 2 onward, tasks where both the center configuration and its neighbor succeed are skipped, and failure-only fill-ins are added only when a round has fewer than contrast pairs (App. A.2).
Contribution of contrastive attribution. Because in-run fill-ins are harder tasks where both configurations failed, we compare the two diagnostic modes under matched conditions:
- Same 108 failures, matched candidate generation and validation (Table 9): contrastive diagnosis achieves 89.8% repair and 85.3% retention, vs. 71.2% and 70.9% for the failure-only fallback.
- End-to-end optimization (Table 2): replacing contrastive attribution in Rounds 2–6 with the failure-only fallback (
Trajectory-only) lowers Test-Normal TGC by 7.14 points (93.45% to 86.31%) and SGC by 12.50 points (89.29% to 76.79%); disabling contrast-pair priority lowers them by 4.76 and 8.93 points.
We will add this breakdown to Table 10 (App. C.2) and clarify the batch filtering in App. A.2.
W2 / Q3: What are the confidence intervals for the differences against the strongest baselines, especially on WebArena and BBH, are the smaller gains consistent across runs, and what does the shaded band in Figure 6 show?
A3: All six paired 95% CIs on WebArena and BBH exclude zero, and CATO achieves the highest 3-run mean in all 18 settings in Table 1 (with all 18 paired 95% CIs over the strongest competing optimizer excluding zero). Because all methods are evaluated on the same held-out test tasks, we compute CIs via a task-level paired bootstrap pooling the three runs (scenario-level for SGC). 3-run mean ± SD:
| Setting | Strongest baseline | CATO | Δ [95% CI] |
|---|---|---|---|
| WebArena SR, GLM-5.2 | 52.12 ± 1.21 (ACE) | 55.76 ± 1.60 | +3.64 [+1.62, +5.86] |
| WebArena SR, Qwen3.7-Plus | 52.73 ± 1.60 (ACE) | 55.15 ± 1.05 | +2.42 [+0.81, +4.24] |
| WebArena SR, DeepSeek-V4-Flash | 50.91 ± 1.82 (ACE) | 52.73 ± 1.21 | +1.82 [+0.61, +3.23] |
| BBH Macro SR, GLM-5.2 | 92.76 ± 0.61 (GEPA) | 93.93 ± 0.38 | +1.17 [+0.60, +1.87] |
| BBH Macro SR, Qwen3.7-Plus | 93.03 ± 0.34 (GEPA) | 94.11 ± 0.26 | +1.08 [+0.53, +1.68] |
| BBH Macro SR, DeepSeek-V4-Flash | 91.50 ± 0.35 (GEPA) | 92.26 ± 0.29 | +0.76 [+0.33, +1.25] |
On AppWorld with GLM-5.2, the margins over GEPA are +2.97 TGC [+1.19, +4.96] and +5.96 SGC [+1.79, +10.12] on Test-Normal and +4.08 SGC [+1.68, +6.71] on Test-Challenge.
BBH. Each run is the macro-average across 10 independently optimized subtasks, which keeps CATO's 3-run SD between ±0.26% and ±0.38%. The BBH margins are smaller because 46.6–63.3% of the attribution score mass falls on the system prompt (Table 10), which GEPA already optimizes directly above 91%, whereas AppWorld failures span all three components.
Shaded band in Figure 6. We agree that the shaded band in Figure 6a does not measure variability across independent runs: it shows the standard deviation across the population members within a single 10-round run. We will state this in the Figure 6 caption (App. D.6), summarize the run-level consistency in Sec. 4.2 and 4.4, and add the full 3-run SD and paired 95% CI tables to App. D.
We thank you again for your thorough reading and constructive suggestions.
Response to Reviewer BUUo (Part 1/2)
We thank the reviewer for the precise and constructive review, and for noting that the fault-injection protocol connects attribution quality to downstream repair utility and that the end-to-end results and resource accounting support targeted editing. We answer each question below with new evaluations and run-level statistics.
W1 / Q1: Under the actual assignment rule of scores greater than 0.6, what are the precision and recall for identifying the injected component, how often are additional components assigned, and what are the uncertainty estimates accounting for shared faulty configurations?
A1: Under the optimization assignment rule A(r)={d:sd(r)>0.6} (Eq. 3; whereas Table 8 reports strict top-1 accuracy), CATO reaches 93.5% recall and 91.8% precision across all 108 cases (three repetitions) and assigns an additional non-faulty component in only 8.3% of cases. The 95% CIs come from a cluster bootstrap that resamples the 12 faulty configurations (3 Prompt, 5 Tools, 4 Experience):
Recall is the fraction of cases in which the faulty component is assigned, and precision is the fraction of assigned components that are faulty.
On your W1 point about naturally occurring failures with several plausible repair locations and interacting defects, we ran two further studies:
We will summarize these evaluations in Sec. 4.3 and add the sd>0.6 metrics, configuration-clustered CIs, and both studies to App. C.1.