- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents LiteEvo, a cost-efficient framework for automatically evolving LLM-agent harnesses without benchmark-specific handcrafted initialization. The method coordinates four tool-free meta-agents—an Analyst, Librarian, Composer, and Editor—to extract reusable components from execution traces, maintain a versioned component library, compose candidate harnesses, and retain revisions through a performance gate. Across five agent benchmarks, the paper reports higher mean pass@2 than a reproduction of HarnessX while reducing the reported meta-agent API cost by approximately 13×. Its inductive protocol evolves on training tasks, selects on validation tasks, and evaluates on unseen test tasks; additional experiments transfer the resulting harness to Claude Code with Sonnet 4.6 and analyze the respective contributions of prompts and processors.
Critical strengths
-
The work directly addresses two consequential limitations of prior harness-evolution systems: reliance on benchmark-specific handcrafted seeds and the high cost of agentic meta-optimization. Starting from a common neutral harness improves the conceptual cleanliness of the comparison, while the small number of tool-free meta-agent calls per round makes the proposed procedure substantially more practical.
-
The evaluation spans five agent benchmarks under an inductive train/validation/test protocol, rather than relying exclusively on the tasks used during evolution. The transfer experiment with a second backbone and the prompt-versus-processor decomposition further clarify what the evolved harness contributes and provide useful evidence that the reported benefits are not confined to a single base model.
Critical weaknesses
-
All reported results appear to rely on a single stochastic evolution run. Because both the evolutionary trajectory and best-round selection are stochastic, the absence of independent repetitions, uncertainty estimates, or significance tests makes it difficult to assess robustness. The transductive results are additionally optimistic because the best round is selected using the same graded tasks on which it is reported.
-
The central comparison and generalization claims are affected by evaluation-specific confounds. The HarnessX reproduction uses evaluator-authored seed harnesses for three benchmarks, so its relative performance may depend materially on those choices. In addition, GAIA evolution can encounter benchmark mirrors or copied answers on the public web, creating a plausible leakage path that weakens the interpretation of the corresponding gain.
Questions for authors
-
Can the authors report at least three independent evolution runs, including the distribution of final-round and selected-round performance, and use a held-out selection set so that model selection and final evaluation are separated?
-
How sensitive is the HarnessX comparison to the three evaluator-authored seed harnesses? For GAIA, can the experiment be repeated in an environment that excludes benchmark mirrors and answer copies, or can the authors otherwise quantify the effect of this potential leakage?
5 comments
On final-round versus selected-round performance
| Benchmark | Setting | Method | 1st run In-loop (Best) | 1st run In-loop (Final) | 1st run Test pass@2 (Best) | 1st run Test pass@2 (Final) | 2nd run In-loop (Best) | 2nd run In-loop (Final) | 2nd run Test pass@2 (Best) | 2nd run Test pass@2 (Final) | 3rd run In-loop (Best) | 3rd run In-loop (Final) | 3rd run Test pass@2 (Best) | 3rd run Test pass@2 (Final) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ALFWorld | Transductive | HarnessX | 94.0 | 93.3 | 93.8 ± 2.2 | 96.3 ± 0.7 | 93.3 | 91.8 | 89.6 ± 0.8 | 92.8 ± 1.1 | 95.5 | 95.5 | 95.3 ± 0.4 | 95.3* ± 0.4 |
| LiteEvo | 96.3 | 92.3 | 96.3 ± 0.7 | 90.5 ± 1.1 | 94.2 | 92.5 | 93.5 ± 0.9 | 93.0 ± 0.9 | 93.4 | 93.4 | 95.3 ± 0.9 | 95.3* ± 0.9 | ||
| Inductive | LiteEvo | 95.0 | 94.8 | 94.8 ± 0.7 | 90.8 ± 0.4 | 93.9 | 93.9 | 92.5 ± 0.7 | 92.5* ± 0.7 | 94.4 | 94.4 | 93.5 ± 0.4 | 93.5* ± 0.4 | |
| WebShop | Transductive | HarnessX | 66.0 | 63.0 | 63.0 ± 3.0 | 62.0 ± 3.0 | 68.0 | 68.0 | 63.7 ± 0.6 | 63.7* ± 0.6 | 68.0 | 62.0 | 63.3 ± 4.5 | 62.0 ± 4.0 |
| LiteEvo | 64.2 | 54.8 | 63.0 ± 0.0 | 56.3 ± 0.6 | 69.7 | 68.3 | 68.3 ± 0.6 | 66.0 ± 1.0 | 66.5 | 64.8 | 66.7 ± 0.6 | 65.0 ± 3.0 | ||
| Inductive | LiteEvo | 52.3 | 45.3 | 70.0 ± 1.0 | 62.0 ± 1.0 | 47.2 | 47.2 | 60.0 ± 3.0 | 60.0* ± 3.0 | 52.3 | 52.3 | 60.3 ± 4.0 | 60.3* ± 4.0 | |
| AppWorld | Transductive | HarnessX | 63.1 | 59.5 | 62.3 ± 0.9 | 61.7 ± 1.2 | 63.7 | 57.1 | 63.3 ± 3.8 | 63.7 ± 1.2 | 61.9 | 61.9 | 61.3 ± 1.6 | 61.3* ± 1.6 |
| LiteEvo | 65.5 | 64.4 | 67.7 ± 0.9 | 59.9 ± 3.0 | 68.2 | 60.3 | 63.9 ± 1.5 | 60.9 ± 2.7 | 55.8 | 55.8 | 62.9 ± 2.1 | 62.9* ± 2.1 | ||
| Inductive | LiteEvo | 63.2 | 63.2 | 65.5 ± 1.2 | 65.5* ± 1.2 | 59.9 | 59.6 | 62.3 ± 0.3 | 62.1 ± 2.8 | 64.9 | 64.9 | 65.1 ± 1.5 | 65.1* ± 1.5 | |
| GAIA | Transductive | HarnessX | 68.0 | 64.1 | 64.4 ± 1.5 | 61.5 ± 1.1 | 68.9 | 58.3 | 62.5 ± 4.6 | 59.9 ± 3.1 | 70.9 | 66.0 | 63.4 ± 2.2 | 65.0 ± 3.5 |
| LiteEvo | 64.6 | 64.6 | 65.2 ± 1.6 | 65.2* ± 1.6 | 62.3 | 55.2 | 62.1 ± 1.7 | 58.9 ± 2.4 | 64.1 | 60.2 | 64.1 ± 1.7 | 61.8 ± 3.9 | ||
| SWE-bench Verified | Transductive | HarnessX | 67.3 | 56.4 | 57.0 ± 6.4 | 55.8 ± 2.8 | 60.0 | 52.7 | 57.0 ± 2.8 | 50.3 ± 2.8 | 60.0 | 54.5 | 55.2 ± 5.8 | 57.6 ± 4.6 |
| LiteEvo | 60.0 | 54.5 | 61.8 ± 4.8 | 57.0 ± 3.8 | 63.6 | 52.7 | 64.8 ± 4.2 | 52.7 ± 3.6 | 72.2 | 61.8 | 63.0 ± 2.8 | 63.6 ± 1.8 | ||
| Inductive | LiteEvo | 67.3 | 63.6 | 69.1 ± 4.8 | 60.0 ± 3.1 | 66.7 | 58.2 | 64.8 ± 1.0 | 63.6 ± 1.8 | 67.3 | 61.8 | 64.8 ± 4.2 | 57.6 ± 2.1 |
To assess the impact of best-round selection on the results, we report both the selected-round and final-round performance for each evolution run in the table above. Runs in which the two rounds coincide are marked with an asterisk (*) in the final-round column.
Among the 18 LiteEvo runs in which the two rounds differ, the selected round achieved higher mean test performance than the final round in 17 runs. The only exception was the third transductive run on SWE-bench, where the difference was small, at 0.6 percentage points. In contrast, for HarnessX, the final round outperformed the selected round on the test set in 5 of the 12 runs in which the two rounds differ.
On held-out selection
| Benchmark | Division | Task pool | Train (evolve) | Val (gate) | Test (grade) | Disjoint by |
|---|---|---|---|---|---|---|
| ALFWorld | Native | ALFRED game files | 134 of 3,553 (train) | 140 (valid-seen, all) | 134 (valid-unseen, all) | Game |
| WebShop | Carved | 6,910 goals, 415 distinct products | 100 (51 products) | 100 (58 products) | 100 (54 products) | Product |
| AppWorld | Native | Released splits | 90 (train, all) | 57 (dev, all) | 168 (test-normal, all) | Task |
| GAIA | Fixed subset | 103 text-only questions | Transductive only (the same 103) | -- | ||
| SWE-bench Verified | Carved | 497 instances, 10 repos | 55 | 55 | 55 | Instance |
Held-out selection is already applied in LiteEvo’s inductive experiments. For all inductive runs, the train/validation/test splits are fixed in advance and are disjoint at the game, product, task, or instance level, as shown in the table above. The meta-agent evolves the harness based on trajectories generated on training tasks, while the gate uses separate rollouts on validation tasks to make accept/hold/revert decisions at each round. The resulting harness is then evaluated on test tasks that the evolution loop has never seen. Further details on evaluation independence are provided in our response to Reviewer pKt5 (“On independence of evaluation”).
Weakness 2 & Question 2
| Benchmark | Seed harness | In-loop pass@2: R0 | In-loop pass@2: Peak | In-loop pass@2: Peak round | In-loop pass@2: R0 → peak | Test pass@2 |
|---|---|---|---|---|---|---|
| ALFWorld | HarnessX-styled seed | 62.7 | 94.3 ± 1.1 | R6 / R5 / R7 | +31.6 | 92.9 ± 3.0 |
| LiteEvo neutral seed | 53.7 | 94.0 | R4 | +40.3 | 90.8 ± 1.7 | |
| WebShop | HarnessX-styled seed | 41.7 | 67.3 ± 1.2 | R6 / R7 / R4 | +25.7 | 63.3 ± 0.3 |
| LiteEvo neutral seed | 12.0 | 65.0 | R4 | +53.0 | 66.0 ± 3.6 | |
| AppWorld | HarnessX-styled seed | 47.4 | 62.9 ± 0.9 | R6 / R5 / R7 | +15.5 | 62.3 ± 1.0 |
| LiteEvo neutral seed | 0.0 | 54.2 | R6 | +54.2 | 51.0 ± 3.6 | |
| SWE-bench Verified | HarnessX official seed | 57.0 | 62.4 ± 4.2 | R1 / R2 / R3 | +5.5 | 56.4 ± 1.0 |
| LiteEvo neutral seed | 50.9 | 56.4 | R3 | +5.5 | 53.3 ± 3.8 |
On the sensitivity of HarnessX to seed harness
Except for GAIA, whose official recipe is fully available, our HarnessX reproduction used reasonably constructed seed harnesses or evolution recipes. We therefore conducted one additional evolution run for each of the other four benchmarks using LiteEvo’s neutral initial harness as the seed, with three test evaluations per run. The results are shown in the table above. The evolution loop, meta-agent, runner, and turn/token budgets were kept identical to the original implementation. For a more stable comparison, we used statistics from the three independent evolution runs requested by the reviewer for the original HarnessX reproduction (with HarnessX-styled or HarnessX official seeds).
Except on AppWorld, the choice of seed harness did not lead to substantial differences in test pass@2. On all three benchmarks, the gap between the two arms falls within one standard deviation of one arm: the gaps on WebShop (2.7 percentage points) and SWE-bench Verified (3.1 percentage points) fall within the neutral arm’s standard deviations (±3.6 and ±3.8, respectively), while the gap on ALFWorld (2.1 percentage points) falls within the HarnessX arm’s standard deviation (±3.0). The direction of the difference is also inconsistent, with the neutral seed actually performing better on WebShop. On AppWorld, the neutral seed started with no knowledge of the required method calls for task termination. HarnessX’s meta-agent did not discover all of these requirements within eight rounds, resulting in lower performance.
These results suggest that the seeds we constructed did not disadvantage HarnessX. On ALFWorld and WebShop, the differences show no consistent direction, while on AppWorld, our seed actually benefited HarnessX (averaging 72.8 vs. 69.3 across the three benchmarks). Had both methods used the same neutral initial harness, the comparison would likely have been more favorable to LiteEvo.
We appreciate the reviewer’s valuable comments and insightful questions, which we address point by point below.
Weakness 1 & Question 1
On more independent evolution runs
To assess the reproducibility of our results, we conducted three independent evolution runs for each setting, with the results reported in the table above. For each run, we evaluated the selected round on the test set three times. Round selection was based on in-loop performance in the transductive setting and validation performance in the inductive setting.
In the transductive setting, LiteEvo achieved higher test pass@2 than HarnessX on all five benchmarks when averaged across the three runs. However, the margin on GAIA was very small, at 0.4 percentage points.
In the inductive setting, although LiteEvo does not access test tasks during evolution, it matched or exceeded transductive HarnessX on all four evaluated benchmarks, including a 0.1 percentage-point gain on WebShop. Thus, across both settings, LiteEvo matched or exceeded HarnessX’s performance while reducing API costs by factors of 6.7–17.3 across benchmarks and approximately 12 overall.