- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies when the components added around a base model, such as skills, harnesses, retries, and inference budget, actually help, rather than measuring only their average effect. Using a large matched factorial experiment (4 models × 4 skill types × 4 harnesses × 3 budgets, across eight benchmarks and about 0.9M trajectories), the authors find that no configuration is universally best. The axes interact substantially, and average gains can hide opposing effects across task subgroups. Root-cause failure labels, assigned before the authors examine which interventions fix them, only partially predict the right fix: retries help execution failures and stronger models help capability failures, but skills do not reliably fix procedural failures. Most failures (59.6%) cannot be fixed by changing any single axis, and simple routers fail to beat always using retries (45.3% vs. an oracle at 58.1%). The authors argue that interventions should be evaluated by which failures they fix, under which configuration, and at what cost.
Critical strengths
-
I am personally impressed by the scale of the design, it controls every axis and runs every single tasks and keeps the full trajectories. This is extremely rare in agent experiments. And I think the ledger, if released, will be a real asset for the community.
-
I am impressed by the paper's honesty. It admits that the Procedural to Skill mapping is not supported, and the learned routers do worse than always using H2, and these facts are not supporting the main hypothesis. In addition, the author acknowledges their weaknesses like floor-effect cells and confidence intervals.
PS. I like the LiveMath example in the paper, I think it reflects perfectly how one meta-design has a gain on one group but will drop on another. This means that we have to examine closely on the components of agent systems.
Critical weaknesses
- I am not convinced by the paper's main argument after reading through the whole paper. My biggest concern is that root-cause label and benchmark identity are essentially the same. Six of the eight benchmarks appear to have only a single root category, so a model that uses the root label may largely be recovering benchmark identity rather than learning something more general about failure type. The within-benchmark analysis helps address this, but it is only possible on three benchmarks, and I think the evidence is still mixed on there. Adding the root label improves AUPRC by 3.69 points, but AUROC changes by only 0.04 points.
I am also concerned about the quality of the root-cause labels themselves. The labels are produced by Qwen3.5-9B, which is also one of the evaluated models and the model used to generate all SkillOpt and Trace2Skill artifacts. There is no human agreement study or other reliability estimate, and 43.5% of failures in the source setting remain unlabeled. Given how central these labels are to the main thesis, I think stronger validation is needed.
- Several measurement choices make the headline effect sizes hard to interpret. The largest harness envelopes come from Amazon SOP, HotpotQA Context, and SQuAD: 48.03, 48.18, and 44.34 points, respectively. These are also the benchmarks with the most floor-effect cells. The appendix explains that much of this behavior is driven by answer-format effects, but Section 4.2, Table 3, and Take-away 2 still present these numbers mainly as harness gains. I think this caveat needs to appear directly in the main text, because otherwise the headline numbers are easy to overread.
Questions for authors
Clarifications
- Did the authors check the root labels by hand? I think it would be helpful to have some annotatons on some trajectories to see if how accurate the Qwen judge is. Also if the root labels are noisy, then Table 23 and Figure 4 might change?
- How do the Execution to H2 comparison change if you do the root-label analysis using some other model? Maybe a larger one than Qwen 3.5 9B?
- For the oracle (58.12%), the action is picked and scored on the same runs, right? If you pick on repeat 1 and score on repeats 2–3, how much of the gap over always-H2 is left? I suspect some of it is just noise.
Suggestions that might change my score
- For SQuAD, Hotpot Context and Amazon, could you rescore with a looser answer-format check? Right now I can't tell how much of the big harness gains is real repair and how much is just fixing the output format.
- Please compare the mapped action against the best other action, ideally at similar cost, instead of the mean of the other three. For Execution failures, Model (46.70) already beats H2 (31.42), which makes Take-away 3 hard to accept as written.
3 comments
Weakness 3
Several measurement choices make the headline effect sizes hard to interpret. The largest harness envelopes come from Amazon SOP, HotpotQA Context, and SQuAD: 48.03, 48.18, and 44.34 points, respectively. These are also the benchmarks with the most floor-effect cells. The appendix explains that much of this behavior is driven by answer-format effects, but Section 4.2, Table 3, and Take-away 2 still present these numbers mainly as harness gains. I think this caveat needs to appear directly in the main text, because otherwise the headline numbers are easy to overread.
Response. Agreed. The caveat was only in the appendix; it is now in the main text. Table 3 flags the floor-affected benchmarks and reports the re-scored envelopes next to the native ones, and Section 4.2 states that the SQuAD and Hotpot Context envelopes reflect the answer interface, not execution repair, while Amazon's gain is mostly substantive.
Question 1
Did the authors check the root labels by hand? I think it would be helpful to have some annotatons on some trajectories to see if how accurate the Qwen judge is. Also if the root labels are noisy, then Table 23 and Figure 4 might change?
Response. Not in the submitted version; we have now done so (details under Weakness 2).
- By hand. The labels are accurate where the main conclusions depend on them: 119 of 120 SimpleQA labels are confirmed, and labels tied to logged errors point to the same kind of fix in 81% of cases. An independent GPT-6-sol judge agrees on SimpleQA (κ = 0.77) but not on LiveMath (κ = 0.08).
- Yes, noisy labels change Table 23 and Figure 4 on LiveMath. We removed LiveMath from these and the other root analyses (in the revision: Table 4, Figure 5, and App. D.6, Tables 38–42). The main-text conclusion is unchanged: Execution/interface→H2 (+9.58) and Capability→Model (+64.54) remain positive, also under propagated label errors.
Question 2
How do the Execution to H2 comparison change if you do the root-label analysis using some other model? Maybe a larger one than Qwen 3.5 9B?
Response. Thank you; we ran it. GPT-6-sol, not an evaluated model, relabeled all model-assisted labels that can affect the Execution row, with the same taxonomy, prompt, and evidence, blind to the originals. Execution→H2 under both label sets (same bootstrap as Table 25):
| Labels | n | Benchmarks | H2 − non-aligned mean [95% CI] | H2 − Trace2Skill [95% CI] |
|---|---|---|---|---|
| Paper (Qwen judge) | 465 | 7 | +6.54 [+2.21, +10.77] | +10.23 [+3.33, +16.18] |
| New (GPT-6-sol judge) | 437 | 7 | +5.04 [−0.78, +11.82] | +5.79 [−3.25, +15.57] |
Removing the benchmark whose labels GPT-6-sol does not reproduce strengthens Execution→H2: +9.58 [+4.77, +13.93] with the original labels and +10.05 [+5.81, +14.08] with the GPT-6-sol labels.
Question 3
For the oracle (58.12%), the action is picked and scored on the same runs, right? If you pick on repeat 1 and score on repeats 2–3, how much of the gap over always-H2 is left? I suspect some of it is just noise.
Response. Yes, and we ran your check. Picking on one repeat and scoring on the other two, the gain over the fixed Trace2Skill + H2 policy shrinks from +12.83 to +3.95 points [+1.58, +6.36] (49.24% vs. 45.29%): about 8.9 points were selection on noise, and a small but real headroom remains. We moved the router analysis to the appendix, where this out-of-sample oracle is the main headroom estimate (App. E.1, Table 48; App. E.4).
Suggestion 1
For SQuAD, Hotpot Context and Amazon, could you rescore with a looser answer-format check? Right now I can't tell how much of the big harness gains is real repair and how much is just fixing the output format.
Response. We did this, re-scoring all saved trajectories with looser checks fixed in advance: for SQuAD and Hotpot Context, whether the response contains a gold answer; for Amazon, a parser that relaxes formatting but not whether the action and its values are correct.
| SQuAD | Hotpot Context | Amazon SOP | |
|---|---|---|---|
| Harness envelope ∆H⁺, native scoring | 44.34 | 48.18 | 48.03 |
| Harness envelope ∆H⁺, looser check | 2.12 | 0.00 | 46.33 |
On SQuAD and Hotpot Context, the harness gain is almost entirely output format; on Amazon, it is mostly substantive, with only about 2% surface formatting. Table 3 (Section 4.2) reports both scorings, with details in App. A.6.2.
Suggestion 2
Please compare the mapped action against the best other action, ideally at similar cost, instead of the mean of the other three. For Execution failures, Model (46.70) already beats H2 (31.42), which makes Take-away 3 hard to accept as written.
Response. Thank you, this is a better comparison. You are right that a stronger model rescues more. Switching the model replaces the agent rather than repairing it, so we treat it as the last resort, considered only when same-model actions cannot repair the failure. Among those actions (rerun, skill, harness, budget), H2 is the strongest on execution failures, computed from the task-level outcomes behind Table 26 with the same paired bootstrap. Procedural failures do not map to a skill (+3.77 [−5.66, +14.47], one benchmark), as Take-away 3 states. We revised Take-away 3 accordingly (Section 4.3).
| Failure pattern | Mapped action | Best same-backbone alternative | Mapped − best alternative [95% CI] | Stronger model (GPT-5.6-sol) | Mapped − model [95% CI] |
|---|---|---|---|---|---|
| Execution/interface → H2 (n = 428, 6 benchmarks) | 33.50 | 18.41 (Trace2Skill) | +15.09 [+7.98, +21.48] | 48.18 | −14.67 [−25.48, −6.62] |
Thank you again for your comments and questions, especially on the overlap between root labels and benchmark identity, on label validation, and on the out-of-sample oracle, which substantially improved the rigor of our claims.
Thank you very much for the thorough review and the encouraging comments. We are glad you found the full-factorial design and the retained trajectories valuable, and we agree with the lesson of the LiveMath example: one component can help one subgroup of tasks while hurting another. The anonymized code is now available at https://anonymous.4open.science/r/agent-intervention-matching-757C/, and the trajectory ledger will be released as stated in the Reproducibility Statement.
Please also see the General Response, which restates the question our study answers and presents the new experiments on the bottleneck of self-evolving skills.
Response. We agree this is the central identification issue, and that the within-benchmark evidence is mixed (AUPRC improves, AUROC does not). We therefore narrowed the claim in Section 4.3 (first paragraph; App. D.6.4, Table 39): root labels carry information beyond benchmark identity where several roots occur within a benchmark, but they do not yet predict, independently of the benchmark, which intervention will help.
Sensitivity analysis on a wider support (new). Filling the previously unassigned roots with the judge's candidate labels (Weakness 2) lets more benchmarks meet the paper's rule (at least two roots with at least 20 failures each). With the identical protocol, root labels then improve all three metrics, including AUROC. Because the added labels are unvalidated, we report this only as a sensitivity analysis (App. D.6.4, Table 39).
Response. We agree that using Qwen3.5-9B in three roles is a limitation. In fact, only 305 of the 2,225 labels (all LiveMath) come from the Qwen judge; the rest come from logged errors, evaluator flags, answer-contract violations, or deterministic answer rules on SimpleQA (App. D.6.1, Table 35).
Reliability: we added two validations.
43.5% unlabeled. These failures had no decisive evidence and are concentrated in Hotpot (1,300). The root-conditioned results describe only the labeled subset, as App. A.6 and D.6 state.
Validation of the main thesis. Redrawing every label from the confusion matrix observed in its stratum and recomputing the main contrasts 2,000 times (Section 4.3; App. D.6.2), both stay positive in every draw: