- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents MICA, an intervention-guided framework for agent-harness self-evolution that addresses interactions among context modifications. The central motivation is that a revision that improves one harness component in isolation may behave differently when combined with other revisions. MICA uses matched execution evidence to diagnose potential composition mismatches and applies coordinated template repair rather than treating individual edits as independent.
The approach is evaluated across five environments with GPT-5.5 and DeepSeek-V4-Flash. The paper reports macro-average task scores of 80.90% and 76.80%, respectively. In addition to comparisons with context-learning and harness-optimization baselines, the paper includes controlled mechanism studies using 20 independent optimization runs per configuration on Telecom and AppWorld Normal. These studies compare failure traces against matched interventions and sequential against paired repair.
Critical strengths
-
The paper addresses an important limitation of component-wise optimization: locally beneficial changes may not remain beneficial when combined. This is more specific than the general objective of finding a better prompt or harness, and it provides a useful conceptual framework for understanding failures during iterative agent improvement.
-
Table 4 uses a controlled comparison of two sources of diagnostic evidence and two repair strategies over 20 optimization seeds. Matched interventions improve outcomes relative to ordinary failure traces, while paired repair also improves performance relative to sequential repair under the tested settings. Table 5 further shows that the jointly revised template pair outperforms the stronger unilateral revision. These experiments give substantially more direct support to the proposed mechanism than headline benchmark scores alone.
Critical weaknesses
-
The four-condition intervention analysis is well suited to evaluating two interacting edits, but more complex harnesses may exhibit dependencies involving three or more components. Such higher-order conflicts need not be detectable from pairwise comparisons. The paper does not sufficiently establish how the method handles these situations or how intervention cost scales with the number of mutually dependent components. This limits the scope of the general co-adaptation claim.
-
The main benchmark tables report one optimization seed per method or configuration, whereas several mechanism experiments use 20 seeds. The latter materially strengthen the paper, but they cannot fully establish the stability of every headline comparison, particularly the relatively small average advantage under DeepSeek-V4-Flash. Furthermore, joint repair outperforming unilateral repair should not automatically be interpreted as evidence of a statistically significant positive interaction between the two mechanisms.
Questions for authors
-
How would the proposed intervention mechanism identify and repair failures involving three or more interacting harness components, especially if no individual pair exhibits a clear regression? Please clarify both the algorithmic extension and its execution cost.
-
Can the authors provide uncertainty estimates or additional optimization-seed evidence for the main five-environment comparisons? This is particularly relevant to the smaller average improvement reported for DeepSeek-V4-Flash.
-
For the factorial mechanism study in Table 4, do the matched-intervention and paired-repair mechanisms exhibit a statistically supported interaction beyond their separate effects? Clarifying this would help distinguish the benefit of combining two individually useful components from a stronger synergy claim.
3 comments
Reply - Part 2
W2 + Q2. Optimization-seed uncertainty in the five-environment comparisons
The original main benchmark tables use one optimization seed per method/configuration. The 20-seed mechanism and efficiency experiments do not substitute for external-baseline replication across all five environments. We therefore also report a three-seed, five-environment comparison. The runs use common per-seed resource caps (not necessarily equal actual resource consumption), and scores are macro-averaged over the same five environments within each seed before calculating their across-seed mean and standard deviation:
\bar{A}_m=\frac{1}{3}\sum_{s=1}^{3}A_{m,s}.$$ | Method | GPT-5.5 mean ± SD (%) | DeepSeek-V4-Flash mean ± SD (%) | |---|---:|---:| | ACE | 73.49 ± 0.32 | 67.33 ± 0.32 | | GEPA | 76.23 ± 0.23 | 71.84 ± 0.09 | | SkillOpt | 75.85 ± 0.20 | 75.03 ± 0.28 | | RHO | 72.19 ± 0.29 | 70.95 ± 0.25 | | **MICA** | **80.78 ± 0.60** | **77.00 ± 0.39** | The comparison with the strongest baseline for each model is clearer when the difference is computed within matched seeds: | Paired macro-average difference | Seed 1 | Seed 2 | Seed 3 | Mean ± seed SD | |---|---:|---:|---:|---:| | GPT-5.5: MICA − GEPA | +4.81 pp | +3.63 pp | +5.20 pp | **+4.55 ± 0.82 pp** | | DeepSeek-V4-Flash: MICA − SkillOpt | +1.72 pp | +2.16 pp | +2.02 pp | **+1.97 ± 0.22 pp** | **All three observed seed-level advantages are positive for both models.** The DeepSeek advantage remains smaller than the GPT-5.5 advantage, but is consistently positive across the three observed seeds. This extends the original single-seed gaps of **+4.82 pp** and **+1.72 pp** (Tables 1–2) to three-seed mean gaps of **+4.55 pp** and **+1.97 pp**.Reply - Part 3
Q3. Do matched interventions and paired repair exhibit a statistically supported interaction?
The additional seed-level analysis supports positive main effects of both mechanisms, but does not establish a statistically significant positive interaction.
Thank you for highlighting this distinction. We conducted a factorial interaction analysis using the complete results from 20 independent optimization seeds for each of the four configurations in Table 4.
For each seed, we computed the difference-in-differences:
I = (M+P − M+S) − (F+P − F+S)
We then applied a two-sided paired-seed t-test to the 20 interaction contrasts, using the optimization seed rather than individual task outcomes as the independent statistical unit.
| Statistic | Telecom | AppWorld Normal |
|---|---|---|
| Matched-evidence main effect | +4.63 pp | +5.73 pp |
| Paired-repair main effect | +2.75 pp | +3.65 pp |
| Factorial interaction | +0.25 pp | +1.64 pp |
| 95% CI for interaction | [−3.87, 4.37] | [−1.28, 4.55] |
| Two-sided p-value | 0.900 | 0.254 |
Both mechanisms have statistically positive average main effects, whereas neither interaction estimate is statistically distinguishable from zero.
MICA still achieves the highest 20-seed mean deployed score among the four configurations, reaching 84.50% on Telecom and 81.07% on AppWorld Normal. However, these results support the separate benefits of matched interventions and paired repair, rather than a stronger claim of statistically significant positive synergy.
Reply - Part 1
We thank the reviewer for these constructive comments.
W1 + Q1. Higher-order composition conflicts and intervention cost
The original MICA implementation is explicitly limited to two-edit programs; pairwise interventions alone cannot guarantee detection of a purely three-way conflict. Sections 3.1–3.4 and Appendices B.1/D.1 set the maximum program length to mmax=2; the compiler rejects dependency chains requiring three edits. A failure that appears only when a,b,c are all present may therefore be invisible to every pairwise comparison in the reported two-edit experiments. We agree that the general co-adaptation claim must be read within this scope.
1. Algorithmic extension: test the complete triple, not only its pairs
For three independent edits, one direct extension is to execute the eight presence/absence configurations, using a fixed legal order and separately restored, matched initial states:
S3=∅,a,b,c,ab,ac,bc,abc.
Let RS be the task-balanced reward for subset S. The third-order contrast is
Iabc(3)=Rabc−Rab−Rac−Rbc+Ra+Rb+Rc−R∅.
This contrast can reveal a non-additive three-way effect even when all pairwise contrasts are near zero. In addition to rewards, the diagnostic model would inspect aligned traces, compiler events, edit activation, and task-constraint violations to formulate an evidence-linked repair hypothesis. A triple must not be excluded solely because its individual pairs show no regression; a bounded higher-order exploration allocation would need to sample eligible triples even without a negative pairwise signal. For order-sensitive edits, selected legal orderings must be evaluated separately. For dependent edits, only legal subsets and prefixes can execute; when some of the eight cells are illegal, the full third-order contrast is not identifiable and the diagnosis must instead use legal conditional comparisons and activation evidence.
Repair is a separate operation from presence/absence diagnosis. Once a legal three-edit structure G is fixed, a proposal revising three templates can be evaluated using the 23=8 original/revised combinations on the same fitting tasks and repeat indices. Every hybrid must preserve its interface and compile successfully. As with the two-edit method, actual measured reward—not the diagnostic model's prediction or the sign of an interaction contrast—determines version retention, followed by frozen screening and independent confirmation. Selective evaluation cannot guarantee finding every higher-order conflict unless the relevant triple is actually tested.
2. Execution cost and additional three-edit evidence
A fixed-order, three-edit presence/absence block requires up to 8 task executions per task/repeat; checking additional orders increases this cost. For comparison, the current independent-pair matched block contains 5 executions (both orders), and a dependent-pair block contains 3 (Section 3.3). For fixed-structure repair with m revised templates, full factorial comparison requires
Nrepair(m)=2m,∣H∣,Rfit
fresh task executions per repair hypothesis, excluding proposal generation, broader structure search, screening, and confirmation. Under the current controlled-study panels (12 Telecom tasks or 18 AppWorld Normal tasks; Rfit=2), moving from two to three templates doubles the per-hypothesis repair cost: 96 to 192 rollouts on Telecom and 144 to 288 on AppWorld Normal. Exhaustive consideration of all m-edit subsets from a library of size B adds a (mB) candidate factor, before legal orders, bindings, and dependencies. This is why we do not claim unrestricted higher-order scalability.
Our additional experiments implement a bounded three-edit extension, including legal three-edit structures and a selective factorial strategy. They use the following larger per-seed caps:
The selective factorial variant exceeds local pair repair by 2.50 and 3.37 percentage points, respectively, under the reported new caps. This demonstrates a feasible, bounded three-edit extension, not a general scaling solution or a guarantee of detecting every pure higher-order conflict. Larger edit sets will require selectively testing suspected interacting subsets, reserving exploration for triples without pairwise warning signals, and validating the resulting complete program.