acceptodds
Under review as a conference paper at ICLR 2027

When Do Agent Interventions Help? Matching Interventions to Residual Failures

Abstract

Modern agent systems wrap a language model in skills, execution harnesses, recovery, reflection, and extra inference budget. These interventions, however, are usually evaluated in isolation or under a fixed configuration, obscuring whether an observed gain reflects intrinsic value or merely a match to the failures the surrounding system happens to leave behind. Rather than asking whether an intervention helps on average, we ask when each one helps and what failure it repairs. We run a matched factorial study over nine benchmarks, four models, four skill conditions, four harness configurations, and multiple inference budgets, for nearly 0.9 million agent trajectories. We find that intervention value is conditional rather than intrinsic: no single configuration is optimal everywhere, and the payoff of model, skill, harness, and budget interventions depends strongly on the task and its surrounding setup. This heterogeneity is structured by the residual failure, the failure that remains after the current system has done its best. Trajectory-level failure labels capture part of this structure, but diagnosis does not guarantee repair: even an observable execution failure need not be recoverable. These conditional patterns define capability boundaries and reveal real headroom for selective intervention. Yet the tested lightweight deployable routers fail to beat a fixed Harness policy on quality and cost, despite a far stronger outcome-aware oracle. Agent interventions should be judged by which failure they repair, under which configuration, and at what cost, not by average component gains alone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agent harness, scaffolding, factorial study, matched comparison, intervention analysis, failure attribution, root cause, skills, test-time compute, agent evaluation, routing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ikjVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when the components added around a base model, such as skills, harnesses, retries, and inference budget, actually help, rather than measuring only their average effect. Using a large matched factorial experiment (4 models × 4 skill types × 4 harnesses × 3 budgets, across eight benchmarks and about 0.9M trajectories), the authors find that no configuration is universally best. The axes interact substantially, and average gains can hide opposing effects across task subgroups. Root-cause failure labels, assigned before the authors examine which interventions fix them, only partially predict the right fix: retries help execution failures and stronger models help capability failures, but skills do not reliably fix procedural failures. Most failures (59.6%) cannot be fixed by changing any single axis, and simple routers fail to beat always using retries (45.3% vs. an oracle at 58.1%). The authors argue that interventions should be evaluated by which failures they fix, under which configuration, and at what cost.

Critical strengths
  1. I am personally impressed by the scale of the design, it controls every axis and runs every single tasks and keeps the full trajectories. This is extremely rare in agent experiments. And I think the ledger, if released, will be a real asset for the community.

  2. I am impressed by the paper's honesty. It admits that the Procedural to Skill mapping is not supported, and the learned routers do worse than always using H2, and these facts are not supporting the main hypothesis. In addition, the author acknowledges their weaknesses like floor-effect cells and confidence intervals.

PS. I like the LiveMath example in the paper, I think it reflects perfectly how one meta-design has a gain on one group but will drop on another. This means that we have to examine closely on the components of agent systems.

Critical weaknesses
  1. I am not convinced by the paper's main argument after reading through the whole paper. My biggest concern is that root-cause label and benchmark identity are essentially the same. Six of the eight benchmarks appear to have only a single root category, so a model that uses the root label may largely be recovering benchmark identity rather than learning something more general about failure type. The within-benchmark analysis helps address this, but it is only possible on three benchmarks, and I think the evidence is still mixed on there. Adding the root label improves AUPRC by 3.69 points, but AUROC changes by only 0.04 points.

I am also concerned about the quality of the root-cause labels themselves. The labels are produced by Qwen3.5-9B, which is also one of the evaluated models and the model used to generate all SkillOpt and Trace2Skill artifacts. There is no human agreement study or other reliability estimate, and 43.5% of failures in the source setting remain unlabeled. Given how central these labels are to the main thesis, I think stronger validation is needed.

  1. Several measurement choices make the headline effect sizes hard to interpret. The largest harness envelopes come from Amazon SOP, HotpotQA Context, and SQuAD: 48.03, 48.18, and 44.34 points, respectively. These are also the benchmarks with the most floor-effect cells. The appendix explains that much of this behavior is driven by answer-format effects, but Section 4.2, Table 3, and Take-away 2 still present these numbers mainly as harness gains. I think this caveat needs to appear directly in the main text, because otherwise the headline numbers are easy to overread.
Questions for authors

Clarifications

  1. Did the authors check the root labels by hand? I think it would be helpful to have some annotatons on some trajectories to see if how accurate the Qwen judge is. Also if the root labels are noisy, then Table 23 and Figure 4 might change?
  2. How do the Execution to H2 comparison change if you do the root-label analysis using some other model? Maybe a larger one than Qwen 3.5 9B?
  3. For the oracle (58.12%), the action is picked and scored on the same runs, right? If you pick on repeat 1 and score on repeats 2–3, how much of the gap over always-H2 is left? I suspect some of it is just noise.

Suggestions that might change my score

  1. For SQuAD, Hotpot Context and Amazon, could you rescore with a looser answer-format check? Right now I can't tell how much of the big harness gains is real repair and how much is just fixing the output format.
  2. Please compare the mapped action against the best other action, ideally at similar cost, instead of the mean of the other three. For Execution failures, Model (46.70) already beats H2 (31.42), which makes Take-away 3 hard to accept as written.
3 comments

Thank you very much for the thorough review and the encouraging comments. We are glad you found the full-factorial design and the retained trajectories valuable, and we agree with the lesson of the LiveMath example: one component can help one subgroup of tasks while hurting another. The anonymized code is now available at https://anonymous.4open.science/r/agent-intervention-matching-757C/, and the trajectory ledger will be released as stated in the Reproducibility Statement.

Please also see the General Response, which restates the question our study answers and presents the new experiments on the bottleneck of self-evolving skills.


Weakness 1

I am not convinced by the paper's main argument after reading through the whole paper. My biggest concern is that root-cause label and benchmark identity are essentially the same. Six of the eight benchmarks appear to have only a single root category, so a model that uses the root label may largely be recovering benchmark identity rather than learning something more general about failure type. The within-benchmark analysis helps address this, but it is only possible on three benchmarks, and I think the evidence is still mixed on there. Adding the root label improves AUPRC by 3.69 points, but AUROC changes by only 0.04 points.

Response. We agree this is the central identification issue, and that the within-benchmark evidence is mixed (AUPRC improves, AUROC does not). We therefore narrowed the claim in Section 4.3 (first paragraph; App. D.6.4, Table 39): root labels carry information beyond benchmark identity where several roots occur within a benchmark, but they do not yet predict, independently of the benchmark, which intervention will help.

Sensitivity analysis on a wider support (new). Filling the previously unassigned roots with the judge's candidate labels (Weakness 2) lets more benchmarks meet the paper's rule (at least two roots with at least 20 failures each). With the identical protocol, root labels then improve all three metrics, including AUROC. Because the added labels are unvalidated, we report this only as a sensitivity analysis (App. D.6.4, Table 39).

SupportFailuresOmnibus permutation pLog-loss reductionAUPRC gain (pp)AUROC gain (pp)
Paper: 3 benchmarks1,3500.0002+0.0040 [−0.0008, +0.0090]+3.69 [+1.69, +5.75]−0.04 [−1.31, +1.26]
Wider support: 5 benchmarks (LiveMath excluded, Weakness 2)2,7820.0002+0.0052 [+0.0021, +0.0083]+2.66 [+0.79, +4.78]+1.63 [+0.93, +2.35]

Weakness 2

I am also concerned about the quality of the root-cause labels themselves. The labels are produced by Qwen3.5-9B, which is also one of the evaluated models and the model used to generate all SkillOpt and Trace2Skill artifacts. There is no human agreement study or other reliability estimate, and 43.5% of failures in the source setting remain unlabeled. Given how central these labels are to the main thesis, I think stronger validation is needed.

Response. We agree that using Qwen3.5-9B in three roles is a limitation. In fact, only 305 of the 2,225 labels (all LiveMath) come from the Qwen judge; the rest come from logged errors, evaluator flags, answer-contract violations, or deterministic answer rules on SimpleQA (App. D.6.1, Table 35).

  • Reliability: we added two validations.

    • Independent stronger judge. GPT-6-sol, not an evaluated model, relabeled all 305 LiveMath and 780 SimpleQA hypothesis labels, blind to the originals. SimpleQA labels are validated (κ = 0.77); LiveMath labels are not reproducible (κ = 0.08), because most LiveMath failures are a bare choice letter.
    • Author verification. An author verified a stratified sample of 200 labels, with an LLM assistant (Claude Opus 5.5) proposing each root blind to the original label. SimpleQA labels are confirmed (99.2%, κ = 0.98), and evidence-linked labels agree at the intervention-axis level (81%).
  • 43.5% unlabeled. These failures had no decisive evidence and are concentrated in Hotpot (1,300). The root-conditioned results describe only the labeled subset, as App. A.6 and D.6 state.

  • Validation of the main thesis. Redrawing every label from the confusion matrix observed in its stratum and recomputing the main contrasts 2,000 times (Section 4.3; App. D.6.2), both stay positive in every draw:

    ContrastObserved [95% CI]Draws above zero
    Execution/interface → H2+9.58 [+4.77, +13.93]100%
    Capability → Model+64.54 [+58.68, +70.16]100%

Weakness 3

Several measurement choices make the headline effect sizes hard to interpret. The largest harness envelopes come from Amazon SOP, HotpotQA Context, and SQuAD: 48.03, 48.18, and 44.34 points, respectively. These are also the benchmarks with the most floor-effect cells. The appendix explains that much of this behavior is driven by answer-format effects, but Section 4.2, Table 3, and Take-away 2 still present these numbers mainly as harness gains. I think this caveat needs to appear directly in the main text, because otherwise the headline numbers are easy to overread.

Response. Agreed. The caveat was only in the appendix; it is now in the main text. Table 3 flags the floor-affected benchmarks and reports the re-scored envelopes next to the native ones, and Section 4.2 states that the SQuAD and Hotpot Context envelopes reflect the answer interface, not execution repair, while Amazon's gain is mostly substantive.


Question 1

Did the authors check the root labels by hand? I think it would be helpful to have some annotatons on some trajectories to see if how accurate the Qwen judge is. Also if the root labels are noisy, then Table 23 and Figure 4 might change?

Response. Not in the submitted version; we have now done so (details under Weakness 2).

  • By hand. The labels are accurate where the main conclusions depend on them: 119 of 120 SimpleQA labels are confirmed, and labels tied to logged errors point to the same kind of fix in 81% of cases. An independent GPT-6-sol judge agrees on SimpleQA (κ = 0.77) but not on LiveMath (κ = 0.08).
  • Yes, noisy labels change Table 23 and Figure 4 on LiveMath. We removed LiveMath from these and the other root analyses (in the revision: Table 4, Figure 5, and App. D.6, Tables 38–42). The main-text conclusion is unchanged: Execution/interface→H2 (+9.58) and Capability→Model (+64.54) remain positive, also under propagated label errors.

Question 2

How do the Execution to H2 comparison change if you do the root-label analysis using some other model? Maybe a larger one than Qwen 3.5 9B?

Response. Thank you; we ran it. GPT-6-sol, not an evaluated model, relabeled all model-assisted labels that can affect the Execution row, with the same taxonomy, prompt, and evidence, blind to the originals. Execution→H2 under both label sets (same bootstrap as Table 25):

LabelsnBenchmarksH2 − non-aligned mean [95% CI]H2 − Trace2Skill [95% CI]
Paper (Qwen judge)4657+6.54 [+2.21, +10.77]+10.23 [+3.33, +16.18]
New (GPT-6-sol judge)4377+5.04 [−0.78, +11.82]+5.79 [−3.25, +15.57]

Removing the benchmark whose labels GPT-6-sol does not reproduce strengthens Execution→H2: +9.58 [+4.77, +13.93] with the original labels and +10.05 [+5.81, +14.08] with the GPT-6-sol labels.


Question 3

For the oracle (58.12%), the action is picked and scored on the same runs, right? If you pick on repeat 1 and score on repeats 2–3, how much of the gap over always-H2 is left? I suspect some of it is just noise.

Response. Yes, and we ran your check. Picking on one repeat and scoring on the other two, the gain over the fixed Trace2Skill + H2 policy shrinks from +12.83 to +3.95 points [+1.58, +6.36] (49.24% vs. 45.29%): about 8.9 points were selection on noise, and a small but real headroom remains. We moved the router analysis to the appendix, where this out-of-sample oracle is the main headroom estimate (App. E.1, Table 48; App. E.4).


Suggestion 1

For SQuAD, Hotpot Context and Amazon, could you rescore with a looser answer-format check? Right now I can't tell how much of the big harness gains is real repair and how much is just fixing the output format.

Response. We did this, re-scoring all saved trajectories with looser checks fixed in advance: for SQuAD and Hotpot Context, whether the response contains a gold answer; for Amazon, a parser that relaxes formatting but not whether the action and its values are correct.

SQuADHotpot ContextAmazon SOP
Harness envelope ∆H⁺, native scoring44.3448.1848.03
Harness envelope ∆H⁺, looser check2.120.0046.33

On SQuAD and Hotpot Context, the harness gain is almost entirely output format; on Amazon, it is mostly substantive, with only about 2% surface formatting. Table 3 (Section 4.2) reports both scorings, with details in App. A.6.2.


Suggestion 2

Please compare the mapped action against the best other action, ideally at similar cost, instead of the mean of the other three. For Execution failures, Model (46.70) already beats H2 (31.42), which makes Take-away 3 hard to accept as written.

Response. Thank you, this is a better comparison. You are right that a stronger model rescues more. Switching the model replaces the agent rather than repairing it, so we treat it as the last resort, considered only when same-model actions cannot repair the failure. Among those actions (rerun, skill, harness, budget), H2 is the strongest on execution failures, computed from the task-level outcomes behind Table 26 with the same paired bootstrap. Procedural failures do not map to a skill (+3.77 [−5.66, +14.47], one benchmark), as Take-away 3 states. We revised Take-away 3 accordingly (Section 4.3).

Failure patternMapped actionBest same-backbone alternativeMapped − best alternative [95% CI]Stronger model (GPT-5.6-sol)Mapped − model [95% CI]
Execution/interface → H2 (n = 428, 6 benchmarks)33.5018.41 (Trace2Skill)+15.09 [+7.98, +21.48]48.18−14.67 [−25.48, −6.62]

Thank you again for your comments and questions, especially on the overlap between root labels and benchmark identity, on label validation, and on the out-of-sample oracle, which substantially improved the rigor of our claims.

Reviewer_JAmPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies when agent interventions help, using a large factorial study over 8 benchmarks (plus BigCode in the appendix), 4 models, 4 skill conditions, 4 harness levels (H0–H3) and 3 inference budgets, with nearly 0.9M trajectories. The authors find that no single configuration is best everywhere, and that the best intervention depends on the task and on the remaining failure (e.g., a stronger model for missing knowledge, a richer harness for execution errors). Finally, they show that their lightweight routers cannot beat simply always using H2.

Critical strengths
  1. The study is large in scale, and the authors honestly report many negative results (e.g., the router does not work, and the procedural failure to skill mapping does not hold).
Critical weaknesses
  1. The benchmark choice does not fit a paper about agent interventions. About half of the benchmarks (SQuAD, Hotpot Context, SimpleQA, LiveMath) are single-turn QA or multiple-choice tasks. Common agent settings such as coding, web or terminal tasks are missing, and the only coding benchmark (BigCode) is moved to the appendix.
  2. The harness design is flawed and not clearly defined. H0 does not tell the model the required answer format, so many H0 scores are near 0. This inflates the largest harness effects (e.g., +40 points on SQuAD and Hotpot Context). Also, it is unclear what “execution feedback and retry” (H2) or “more turns” (Budget) means for single-turn tasks.
  3. The main findings are mostly expected. “No configuration is best everywhere”, “missing knowledge needs a stronger model” and “execution errors need feedback and retry” are close to common sense. After nearly 0.9M trajectories, the paper does not give a practical method, since the routers cannot beat always using H2.
Questions for authors
  1. How is each harness level implemented for each benchmark? What do H1, H2 and H3 mean for single-turn tasks such as SQuAD and SimpleQA?
  2. Why does H0 not tell the model the answer format? How do the results change if all harness levels use the same answer format, or with a relaxed answer evaluator?
  3. Which model generates the Direct skills, and what prompt is used?
  4. Why choose SQuAD and HotpotQA instead of more recent agent benchmarks (e.g., SWE-bench, Terminal-Bench)?
5 comments

Thank you for your review, and for recognizing the scale of the study and our honest reporting of negative results. Please also see the General Response, which restates the question our study answers and presents the new experiments on the bottleneck of self-evolving skills.

Weakness 1

The benchmark choice does not fit a paper about agent interventions. About half of the benchmarks (SQuAD, Hotpot Context, SimpleQA, LiveMath) are single-turn QA or multiple-choice tasks. Common agent settings such as coding, web or terminal tasks are missing, and the only coding benchmark (BigCode) is moved to the appendix.

Response. Our goal is not to rank agents on long-horizon tasks but to ask when each intervention helps and which failure it repairs. This requires benchmarks whose dominant bottlenecks differ, so that the same intervention can be seen helping, saturating, or backfiring (Sec. 3.1, Table 1).

Half of the suite is agent tasks with tools, multiple turns, and changing state: Spreadsheet, UCSB SOP, Amazon SOP, and Hotpot FullWiki. The QA benchmarks are deliberate controls. SimpleQA isolates missing knowledge (only the model moves it: ∆M = 60.85, ∆S⁺ = 0.13); SQuAD isolates the answer-interface contract; Hotpot Context and FullWiki pose the same questions with and without the evidence. We agree that a coding or terminal benchmark is missing. The full factorial on SWE-bench or Terminal-Bench is beyond our compute budget, and a reduced version would not support matched comparisons. BigCodeBench-Hard stays in the appendix because its evaluator is not comparable across models; the main text now points to it with this reason (Section 3.1).


Weakness 2

The harness design is flawed and not clearly defined. H0 does not tell the model the required answer format, so many H0 scores are near 0. This inflates the largest harness effects (e.g., +40 points on SQuAD and Hotpot Context). Also, it is unclear what "execution feedback and retry" (H2) or "more turns" (Budget) means for single-turn tasks.

Response. Thank you for raising this. We keep the H0 design, but we agree the main text did not explain how to read its numbers, and we fixed that.

The answer format at H0 is deliberate. H0 is the benchmark's own prompt with the skill appended; H1 adds the standardized answer interface. Separating them is what makes the interface's contribution measurable.

The real issue is interpretation. On SQuAD and Hotpot Context, the 44–48-point "harness gain" does not mean the harness repairs execution: the model already knows the answer at H0 but does not give it in the required format. Relaxed rescoring confirms this (Question 2). The gain is a real harness effect, but of the interface, not of execution repair.

H2 and Budget on single-turn tasks. H2 feeds observable execution errors back for up to three retries and never sees the evaluator outcome or gold answer; single-turn tasks have no execution errors, so H2 only re-asks when there is no valid <answer> field. Budget raises the token, turn, and timeout ceilings; on single-turn tasks, the turn ceiling only caps the number of H2 re-asks.


Weakness 3

The main findings are mostly expected. "No configuration is best everywhere", "missing knowledge needs a stronger model" and "execution errors need feedback and retry" are close to common sense. After nearly 0.9M trajectories, the paper does not give a practical method, since the routers cannot beat always using H2.

Response. We agree that several findings sound intuitive at a high level. Our contribution is testing them under a matched design, and several did not hold as commonly assumed:

  • Self-evolving skills can learn dataset regularities instead of procedures. On LiveMath, one answer option recurs and is systematically correct when present. For Sonnet 5, Trace2Skill gains 6.60 points on the 53 items with this option and loses 4.93 on the other 71, a net effect of about zero (Fig. 3d).
  • Self-evolution cannot supply missing knowledge on its own. On SimpleQA, only a stronger model moves the score. In a new experiment (General Response §2), the same Qwen3.5-9B rises from 5.0% to 60.0% when a short human-written skill makes it request the missing fact and an external source answers.
  • Effects interact rather than add. On SQuAD, a skill that supplies the answer format makes H1 largely redundant (Question 2). Adding interaction terms improves held-out prediction on 7 of 8 benchmarks (leave-one-cell-out R² 0.729 → 0.812; sign-flip p = 0.0078).

A less expected finding. Self-evolving methods often fail to improve on a short human-written skill (General Response §1–§3), so self-evolution does not yet remove the need for a human in the loop. This is now the main lesson of the skill axis in Section 5.

On a practical method. We agree that no router beats the fixed Trace2Skill + H2 policy, and we moved the router to the appendix (App. E).

Question 1

How is each harness level implemented for each benchmark? What do H1, H2 and H3 mean for single-turn tasks such as SQuAD and SimpleQA?

Response. H0 is the benchmark's own prompt with the skill appended. H1–H3 differ by benchmark:

BenchmarkH1 addsH2 feeds back (≤3 retries)H3 (one outcome-blind pass)
SQuAD / Hotpot Context / SimpleQA<answer> contractFixed re-ask, only if no valid <answer>Reflection; answer replaced only if it parses
LiveMathStandardized skill package (answer label already required at H0)Re-ask if the label is not a valid choiceReflection, if turns remain
SpreadsheetStandardized skill packageCode execution errorReflection over the code and its output or error
UCSB SOPStandardized skill packageNative tool errorsTrajectory reflection, then replay from reset
Amazon SOPStandardized skill packageRejected or failed actionsReflection, then replay from reset
Hotpot FullWikiStandardized skill packageTool exception text (e.g., unknown title)Reflection

On single-turn tasks, H1 adds the answer interface, H2 re-asks only when it is not satisfied, and H3 always adds one revision that can change the answer, seeing neither evaluator outcomes nor gold answers (App. A.1.3, Table 12).


Question 2

Why does H0 not tell the model the answer format? How do the results change if all harness levels use the same answer format, or with a relaxed answer evaluator?

Response. Thank you for suggesting the relaxed evaluator, which we ran.

Why. Each harness level adds one component, so each contribution can be measured. If H0 already specified the format, H0 and H1 would coincide on these benchmarks. LiveMath illustrates this: its native prompt already requires the answer label, and its harness envelope is near zero (∆H⁺ = 0.79). We agree that H0→H1 on SQuAD and Hotpot Context therefore mostly measures format specification.

Skills and the harness overlap (Table 2). A skill can take over H1's role: at H0 on SQuAD, the one-line human-written initial skill raises Qwen3.5-9B from 10.7 to 76.9. A skill can also break the format: with SkillOpt, GPT-5.6-sol drops from 62.0 to 13.6 on Hotpot Context at H1.

Relaxed rescoring (new). We re-scored all 115,200 saved SQuAD and Hotpot Context trajectories with evaluators fixed in advance; the most lenient, answer content present, checks only whether the response contains a gold answer. The harness envelope is almost entirely answer format: judged on content, it falls from 44.34 to 2.12 on SQuAD and from 48.18 to 0.00 on Hotpot Context. A manual check found 92–97% of relaxed credits correct. On Amazon SOP, by contrast, about half of the harness gain is substantive repair.

BenchmarkScoringH0H1H2H3∆H⁺
SQuADNative EM36.6576.1780.3180.9944.34
Token F160.4091.4391.2591.6431.24
Answer content present93.1794.6294.4795.292.12
Hotpot ContextNative EM12.1758.9058.9960.3548.18
Token F134.6574.3674.5776.0141.37
Answer content present76.8273.9574.2476.790.00

Rows: Native EM is the paper's exact-match scorer; Token F1 gives partial credit for word overlap; Answer content present credits a response of at most 100 tokens that contains a gold answer. Values are harness marginals (0–100), averaged over models and skills; ∆H⁺ is the harness envelope as in Table 3. Full tables are in App. A.6.2 (Tables 23–24), and Table 3 and Section 4.2 now report both scorings.


Question 3

Which model generates the Direct skills, and what prompt is used?

Response. DIRECT skills are generated by Qwen3.5-9B in one call (temperature 0, thinking disabled, seed 42), with no rollouts, feedback, or validation gate (App. A.1.2, Table 8). The input is a frozen catalog of training-split items only. (App. B wrongly listed 24 Spreadsheet method-validation items as part of this catalog; we corrected it (App. B; App. A.1.2, Table 9).) The shared prompt reads:

"Create one concise, reusable domain skill in Markdown from the frozen training catalog below. Do not solve or enumerate individual examples. Synthesize general procedures, checks, output discipline, and common failure avoidance. This is a single Direct-Synthesis call: no rollout, execution feedback, validation gate, retrieval, or later update. Return only the Markdown skill."

It is used for SimpleQA, SQuAD, Hotpot, and Amazon SOP (with the official SOP text appended); Spreadsheet, LiveMath, and UCSB SOP use benchmark-specific variants. All prompts and catalogs are in App. A.1.2 and the released code.

Question 4

Why choose SQuAD and HotpotQA instead of more recent agent benchmarks (e.g., SWE-bench, Terminal-Bench)?

Response. Our aim is to observe interventions across distinct bottlenecks. SQuAD and HotpotQA are both QA, but they need different fixes:

  • SQuAD isolates the answer-interface contract; the matching fix is H1.
  • Hotpot Context tests multi-hop integration when the evidence is supplied.
  • Hotpot FullWiki asks the same questions but adds retrieval and long horizons, and the best fix shifts to Skill and Budget.

This matched pair lets us attribute a change in the helpful intervention to the change in task structure; SWE-bench and Terminal-Bench mix many bottlenecks within each task. We agree they would strengthen external validity, but as noted under Weakness 1, their full factorial exceeds our compute budget.


Thank you again for these detailed questions, which helped us clarify the harness definitions and the answer-format caveat.

Reviewer_Eht5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses the issue that agent interventions such as skills or test-time-compute scaling are often evaluated in isolation, which does not generalize to all scenarios. They run an exhaustive "factorial study", i.e., trying out all potential combinations of interventions in various benchmarks, showing that the success of an intervention is conditional on other factors and cannot be investigated in isolation.

Critical strengths
  • The evaluation is comprehensive and shows how the performance is affected by changing exactly one of the parameters (model, skill, harness, budget).
  • The finding that no configuration is globally optimal is interesting and well-supported.
Critical weaknesses
  • Prior work on this topic, like SkillsBench [1], which has investigated over 2 million skills, and various harnesses and models, is not discussed or positioned against.
  • The code has not been made available in contradiction to the reproducibility statement, which prevents understanding many key details. For example, it's unclear how the harnesses and skills were constructed, how task errors were labeled, or what "Stage-A" of the router is.
  • The takeaways do not seem justified in their current form. Takeaway 2 says that the task bottlenecks already indicate what the likely bottleneck will be, and takeaway 3 says that matching the failure with the fix remedies the issue. This seems overstated based on the results in Table 26. They show that a stronger model always increases performance most. And, for example, for capability-limited tasks, the skill helps more on the capability-limited tasks (5.51 to 31.60) than on the procedural-limited tasks (9.43 to 20.75), where the work says that skills are the best solution.

[1]: Li, X., Liu, Y., Chen, W., You, B., Di, Z., He, Y., ... & Song, D. (2026). SkillsBench: Benchmarking how well agent skills work across diverse tasks. arXiv preprint arXiv:2602.12670.

Questions for authors
  • How does this work compare to prior work like SkillsBench or work on LLM routers?
  • Are the Trace2Skill trajectories only from successful trajectories? And why would a stronger model like Sonnet 5 benefit from the Qwen3.5-9B trajectory? Figure 3 seems to indicate that the skills are overall not very useful.
  • Have the Qwen labels been cross-checked with human annotators? How high is the agreement?
  • What is the value of the router, since the results indicate it performs poorly?
2 comments

We thank the reviewer for the careful review, and for recognizing that the evaluation is comprehensive and that "no configuration is globally optimal" is well supported.Please also see the General Response, which restates the question our study answers and presents the new experiments on the bottleneck of self-evolving skills.

Weakness 1

Prior work on this topic, like SkillsBench [1], which has investigated over 2 million skills, and various harnesses and models, is not discussed or positioned against.

Response. Thank you for pointing us to SkillsBench (Li et al., 2026). We added it to Related Work (Section 2).

  • What SkillsBench does. It compares no skill, curated (human-written) skills, and self-generated skills, and finds that curated skills help on average while self-generated skills do not.
  • How our study differs. We evaluate self-evolving skills, which SkillsBench does not: SkillOpt optimizes a skill from scored rollouts, and Trace2Skill distills it from successful and failed rollouts. We compare both with a one-shot self-generated skill (DIRECT, close to SkillsBench's setting) and with no skill, varying one axis at a time across model, harness, and budget (General Response §1).

Weakness 2

The code has not been made available in contradiction to the reproducibility statement, which prevents understanding many key details. For example, it's unclear how the harnesses and skills were constructed, how task errors were labeled, or what "Stage-A" of the router is.

Response. We apologize. The code is now available at https://anonymous.4open.science/r/agent-intervention-matching-757C/, and full descriptions are in the revised appendix (App. A.1 for harnesses and skills, App. D.6.1–D.6.2 for labels, App. E for the router).

  • Harnesses. H0 is the benchmark's own prompt with the skill appended. H1 adds a standardized answer or tool interface. H2 adds error feedback with up to three retries. H3 adds one revision that sees neither the evaluator outcome nor the gold answer (App. A.1.3, Table 12).
  • Skills. All skills are built by Qwen3.5-9B on the training split and frozen before testing. DIRECT is one call on a catalog of training items. SkillOpt optimizes from scored rollouts. Trace2Skill distills rollouts and keeps the evolved skill only if it strictly beats its initial skill on method validation (App. A.1.2, Tables 8–10; App. B).
  • Error labels. Each failed trajectory receives one root label from an eight-category taxonomy (Table 37), frozen before any matched outcome is seen. Labels come from logged errors, evaluator flags or contract violations, deterministic answer rules (SimpleQA), or a Qwen3.5-9B judge (LiveMath) (App. D.6.1, Table 35).
  • Router. Stage A is the predictor: a ridge regression predicts, from pre-outcome inputs only, whether each action beats the reference (Trace2Skill + H2), with nested leave-one-benchmark-family-out cross-validation. Stage B picks the action with the highest predicted gain (App. E).

Weakness 3

The takeaways do not seem justified in their current form. Takeaway 2 says that the task bottlenecks already indicate what the likely bottleneck will be, and takeaway 3 says that matching the failure with the fix remedies the issue. This seems overstated based on the results in Table 26. They show that a stronger model always increases performance most. And, for example, for capability-limited tasks, the skill helps more on the capability-limited tasks (5.51 to 31.60) than on the procedural-limited tasks (9.43 to 20.75), where the work says that skills are the best solution.

Response. Thank you for this careful reading. We agree that Take-away 2 was worded too strongly and revised it (Section 4.2).

A stronger model wins every row of Table 26. Switching the model replaces the agent rather than repairing it, so we treat it as the last resort. Among actions that keep the model (rerun, skill, harness, budget), H2 is the strongest on execution failures:

Failure patternMapped actionBest same-backbone alternativeMapped − best alternative [95% CI]
Execution/interface → H2 (n = 428, 6 benchmarks; LiveMath removed)33.5018.41+15.09 [+7.98, +21.48]

The capability row (5.51 → 31.60) pools LiveMath and SimpleQA. Split by benchmark:

Capability row (Qwen3.5-9B failures)RerunSkill (Trace2Skill)
LiveMath, all10.661.7
– meta-option items6.181.8
– non-meta items21.414.3
SimpleQA0.41.5
Benchmark-equal (Table 26)5.5131.60

The skill gain comes from a shortcut: on LiveMath's meta-option items, the skill learned a rule for picking answers (Section 4, Fig. 3d; App. D.3.1). On SimpleQA, it rescues only 1.5% of failures. Take-away 2 now says that the bottleneck is informative about (not determinative of) the axis with traction (Section 4.2).

Question 1

How does this work compare to prior work like SkillsBench or work on LLM routers?

Response.

  • SkillsBench. See Weakness 1: it evaluates curated and self-generated skills, not self-evolving ones.
  • LLM routers. RouteLLM, RouterBench, and FrugalGPT choose which model answers a query; we keep the model fixed and choose which intervention to apply. We added this comparison to Related Work (Section 2).

Question 2

Are the Trace2Skill trajectories only from successful trajectories? And why would a stronger model like Sonnet 5 benefit from the Qwen3.5-9B trajectory? Figure 3 seems to indicate that the skills are overall not very useful.

Response. Thank you for these questions.

Trajectories. Not only successful ones. Following Ni et al. (2026), Trace2Skill rolls out Qwen3.5-9B on the training split and uses both successful and failed trajectories.

Why would Sonnet 5 benefit? Mostly, it uses nothing learned from them. The evolved skill is kept only if it strictly beats its short human-written initial skill on method validation, which happened on only 2 of 7 benchmarks (Spreadsheet and LiveMath); elsewhere the frozen artifact is the human-written skill (Table 10, App. A.1.2). Even on those two, Sonnet 5 does not benefit: on LiveMath its gains on meta-option items are offset by losses elsewhere (Fig. 3d), and on Spreadsheet it scores below NoSkill (Table 25). The original paper reports transfer across Qwen sizes, not from a small Qwen constructor to stronger models of other families; our results suggest such transfer is not guaranteed. A stronger constructor helps: when GPT-5.6-sol builds the skill, Trace2Skill beats the human-written skill on UCSB SOP (+5.38 [+1.71, +9.14]; General Response §2).

Figure 3. It illustrates the heterogeneity of Section 4.1: skills help in some settings (a skill envelope of 10.6 on Hotpot FullWiki) and not in others. You are right that self-evolved skills are limited: in Panel (c), SkillOpt and Trace2Skill score below NoSkill on Qwen Spreadsheet at H0, H2, and H3. We added this analysis to Section 5 (General Response §1–§3).

Question 3

Have the Qwen labels been cross-checked with human annotators? How high is the agreement?

Response. Thank you for this important question. The submitted paper reports no agreement study. We added two validations (App. D.6.2):

  • Author verification (new). An author verified a stratified random sample of 200 labels, with an LLM assistant (Claude Opus 5.5) proposing each root blind to the original label. SimpleQA labels are confirmed (99.2%, κ = 0.98), and evidence-linked labels agree at the intervention-axis level (81%). Propagating the estimated label errors to all labels, both main contrasts stay positive in every draw.
  • Independent stronger judge (new). GPT-6-sol, not an evaluated model, relabeled all model-assisted cases blind to the original labels. SimpleQA labels are validated (κ = 0.77); LiveMath labels are not reproducible (κ = 0.08), because most LiveMath failures are a bare choice letter. We therefore removed LiveMath from all root-conditioned analyses (Table 4, Figure 5; App. D.6, Tables 38–42). Without it, both main contrasts hold: Execution/interface→H2 +9.58 [+4.77, +13.93] and Capability→Model +64.54 [+58.68, +70.16].

Question 4

What is the value of the router, since the results indicate it performs poorly?

Response. Thank you for this question. We agree, and we moved the router to the appendix (App. E, Tables 48–51) so the main text can focus on the matrix results and key case studies. It is a negative but informative result: the oracle's headroom over the fixed policy is real, but cheap signals (task text, a predicted root) do not capture it, and paying for a probe costs more than it gains.


Thank you again for these comments and questions, which helped us make the take-aways more accurate and the paper more reproducible.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response: Self-Evolution Bottleneck, New Experiments, and Paper Revisions

We thank all three reviewers for their careful and constructive reviews. Before the individual replies, we want to restate the question our study answers, why it needs a matched factorial design, and what we found, including a finding from our provenance check that sharpens the conclusion.

A note on the revised paper. The Take-away boxes repeated the section titles, so we removed them: Take-aways 1 and 2 are now the titles of Sections 4.1 and 4.2, and Take-aways 3 and 4 are stated in Section 4.3. We also added a new Section 5, which examines the skill axis on its own: where self-evolved skills fall short and why. The boxes now state the three main findings listed in the Introduction. References to a Take-away in these replies point to that section.

1. What we test, and why it needs a matched matrix

We evaluate self-evolving skills, which differs from SkillsBench. SkillsBench (Li et al., 2026) asks whether having a skill helps. It compares no skill, curated (human-written) skills, and self-generated skills. Our study asks a different question: do self-evolving skill methods work? We test two representative methods, SkillOpt and Trace2Skill, and compare both with a self-generated one-shot skill (DIRECT) and with no skill.

Self-evolving methods are not evaluated under a common standard. Each method paper reports gains under its own evaluation setup, so a gain cannot be attributed to the skill rather than to the surrounding system, and results across papers cannot be compared.

We therefore run a matched factorial matrix. Every frozen skill is evaluated on locked test splits across 8 benchmarks, 4 models, 4 harness levels, and 3 budgets, with 3 repeats per cell.

Result: the matched design reveals three ways self-evolved skills fail:

  • Self-evolved skills cannot supply missing facts. On SimpleQA, the evolved skills leave accuracy unchanged (|gain| ≤ 0.13), because the agent's own experience does not contain the missing facts. The knowledge has to come from outside: from a human, or from an information-seeking mechanism (§2).
  • Evolved skills can learn dataset shortcuts instead of procedures. On LiveMath, the evolved Trace2Skill artifact learned a dataset-specific answer-selection rule rather than a reusable procedure. It helps on the items where the rule happens to hold and hurts on the rest, so the benchmark average hides it (App. D.3.1, Fig. 3d). This suggests that imbalance in the training data can steer self-evolution toward shortcuts.
  • Optimized skills may not transfer across models. On Hotpot Context, the SkillOpt skill optimized on Qwen3.5-9B causes a format problem on GPT-5.6-sol, and its score drops sharply. Our new rescoring shows that GPT-5.6-sol still knows the answer; it just does not submit it in the required format (App. A.6.2).

These failures suggest that self-evolution has a bottleneck.

2. Three hypotheses for the bottleneck, and what the evidence says

We test three hypotheses: (1) the constructor (Qwen3.5-9B) is too weak; (2) the optimization or distillation procedure is the limit; (3) the needed information is missing from the agent's own experience.

(1) Partly supported, but not sufficient (new experiment). We rebuilt SkillOpt and Trace2Skill on UCSB SOP and SimpleQA with GPT-5.6-sol as the constructor, keeping data, initial skills, and gates unchanged, and evaluated them with the paper's protocol, so the scores are comparable with Table 25 (App. C.1, Table 27).

BenchmarkHuman-written skillQwen-built SkillOptGPT-5.6-sol-built SkillOptGPT-5.6-sol-built Trace2SkillGPT-5.6-sol Trace2Skill − human-written skill [95% CI]GPT-5.6-sol SkillOpt − Qwen SkillOpt [95% CI]
UCSB SOP62.0065.3359.2967.38+5.38 [+1.71, +9.14]−6.05 [−10.24, −1.90]
SimpleQA6.757.477.426.85+0.09 [−1.87, +1.97]−0.06 [−1.86, +1.58]

The stronger constructor lets Trace2Skill beat the human-written skill on UCSB SOP but makes SkillOpt worse, and leaves SimpleQA unchanged. Whether it helps therefore depends on the procedure, consistent with SkillsBench (Li et al., 2026).

(2) Partly supported. With the Qwen3.5-9B constructor fixed, a one-shot DIRECT skill beats the iteratively optimized SkillOpt skill on 4 of 8 benchmarks, and on three of them SkillOpt falls below NoSkill:

BenchmarkNoSkillDIRECT (one shot)SkillOpt (iterative)DIRECT − SkillOpt [95% CI]
Hotpot Context50.1451.7736.66+15.11 [+13.51, +16.73]
SQuAD59.4771.7366.33+5.40 [+4.13, +6.69]
Spreadsheet34.7833.1529.91+3.24 [+2.31, +4.15]
Hotpot FullWiki27.3124.2822.63+1.65 [+0.44, +2.83]

(3) Supported on SimpleQA (new experiment; Qwen3.5-9B, H1, 300 test items; App. C.4, Table 30). No skill moves SimpleQA (∆S⁺ = 0.13), whereas a stronger model does (∆M = 60.85).

ArmSkill and knowledge accessAccuracyΔ vs. NoSkill [95% CI]Wrong answers
NoSkillnone5.0—263
Own-data skillall 90 train/method-validation facts6.3+1.3 [−1.3, +3.7]251
Knowledge-boundary skillshort human-written procedure: recognize a missing fact and request it instead of guessing2.0−3.0 [−5.3, −1.0]45
Knowledge-boundary skill + search on requestthe same skill; the requested information is supplied from the items' cited sources60.0+55.0 [+49.0, +60.7]54
External-knowledge skill (oracle)notes from the test items' sources written into the skill (fills the prompt budget)43.3+38.3 [+32.7, +44.0]96
External-knowledge skill, typical sizethe same notes cut to a typical skill size (85 items)13.0+8.0 [+4.0, +12.7]44

A skill built from the agent's own data does not help. A human-written skill that makes the model ask for missing facts stops most wrong guesses, and once the request is answered the same model reaches 60.0%. Knowledge written into a static skill helps only as far as it fits in context.

Implication. Self-evolution from the agent's own experience is not enough: the procedure still needs human authorship or oversight (2), and the missing information has to come from outside (3). We added these analyses as a new Section 5: self-evolving skills do not yet remove the need for a human in the loop.

3. Self-evolved skills fall short of human-written ones

Whatever its cause, the bottleneck has a clear consequence: self-evolution often does not improve on a short human-written skill. In 5 of the 7 main-benchmark Trace2Skill runs, the Qwen3.5-9B-evolved candidate did not beat its starting point on method validation, a short human-written skill (17–85 words). So the frozen "Trace2Skill" artifact on those benchmarks is the human-written skill. We now state this in the paper (Section 5.1; App. A.1.2, Table 10). The table below gives the gate scores on the small method-validation split (0–1); the test-split comparison follows in the second table.

BenchmarkInitial (human) skillEvolved candidate
UCSB SOP0.400.35
Hotpot0.550.50
SQuAD0.800.75
Amazon SOP0.000.00
SimpleQA0.100.10

We also compare on the larger locked test split. There, on the same tasks, the human-written skill matches or beats both the self-generated DIRECT skill and the self-evolved SkillOpt skill on the QA benchmarks. We use SkillOpt because on these benchmarks the Trace2Skill artifact is the human-written skill itself.

BenchmarkHuman seed − NoSkillHuman seed − SkillOptHuman seed − DIRECT
Hotpot FullWiki+10.58 [+8.77, +12.36]+15.25 [+13.23, +17.38]+13.60 [+11.72, +15.65]
SQuAD+17.13 [+15.82, +18.44]+10.27 [+8.53, +12.11]+4.87 [+3.42, +6.36]
Hotpot Context+1.71 [+0.97, +2.41]+15.19 [+13.59, +16.76]+0.08 [−0.73, +0.85]
SimpleQA−0.12 [−0.70, +0.51]−0.24 [−1.09, +0.60]+1.72 [+0.67, +2.74]
Amazon SOP−2.92 [−5.03, −0.99]−2.83 [−4.64, −1.11]+10.37 [+8.70, +12.17]
UCSB SOP+1.89 [+0.74, +3.10]−3.65 [−5.13, −2.19]+1.93 [+0.80, +3.02]

4. Code and new results

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.