acceptodds
Under review as a conference paper at ICLR 2027

LiteEvo: Automated, Cost-Efficient Harness Evolution for Generalization to Unseen Tasks

Abstract

An LLM agent is defined by two things: the weights inside its model and the harness of components assembled around it. Harnesses are still handcrafted, and HarnessX, which evolves them automatically, starts each benchmark from a handcrafted harness, reports gains on the tasks it evolved on, and budgets 100 to 175 million meta-agent tokens per benchmark. We propose LiteEvo, a lightweight harness-evolution algorithm whose tool-free meta-agents mine agent trajectories for reusable components, curate them into a versioned library, and compose each round's harness from it, starting every benchmark from the same neutral harness and never naming the benchmark. Evolving on the graded tasks of five agentic benchmarks with a frozen Qwen3.5-9B, LiteEvo lifts pass@2 by 10.5 to 67.7pp and reaches comparable or higher pass@2 than a reproduction of HarnessX (71.0 against 67.3 on average) at 13.0 lower mean API cost. Harnesses evolved on train tasks keep their gains on unseen test tasks of four benchmarks, and LiteEvo also lifts Claude Code with Sonnet 4.6 by 1.2 to 71.4pp.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
agentic system, evolvable harness, loop engineering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pWceOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents LiteEvo, a cost-efficient framework for automatically evolving LLM-agent harnesses without benchmark-specific handcrafted initialization. The method coordinates four tool-free meta-agents—an Analyst, Librarian, Composer, and Editor—to extract reusable components from execution traces, maintain a versioned component library, compose candidate harnesses, and retain revisions through a performance gate. Across five agent benchmarks, the paper reports higher mean pass@2 than a reproduction of HarnessX while reducing the reported meta-agent API cost by approximately 13×. Its inductive protocol evolves on training tasks, selects on validation tasks, and evaluates on unseen test tasks; additional experiments transfer the resulting harness to Claude Code with Sonnet 4.6 and analyze the respective contributions of prompts and processors.

Critical strengths
  1. The work directly addresses two consequential limitations of prior harness-evolution systems: reliance on benchmark-specific handcrafted seeds and the high cost of agentic meta-optimization. Starting from a common neutral harness improves the conceptual cleanliness of the comparison, while the small number of tool-free meta-agent calls per round makes the proposed procedure substantially more practical.

  2. The evaluation spans five agent benchmarks under an inductive train/validation/test protocol, rather than relying exclusively on the tasks used during evolution. The transfer experiment with a second backbone and the prompt-versus-processor decomposition further clarify what the evolved harness contributes and provide useful evidence that the reported benefits are not confined to a single base model.

Critical weaknesses
  1. All reported results appear to rely on a single stochastic evolution run. Because both the evolutionary trajectory and best-round selection are stochastic, the absence of independent repetitions, uncertainty estimates, or significance tests makes it difficult to assess robustness. The transductive results are additionally optimistic because the best round is selected using the same graded tasks on which it is reported.

  2. The central comparison and generalization claims are affected by evaluation-specific confounds. The HarnessX reproduction uses evaluator-authored seed harnesses for three benchmarks, so its relative performance may depend materially on those choices. In addition, GAIA evolution can encounter benchmark mirrors or copied answers on the public web, creating a plausible leakage path that weakens the interpretation of the corresponding gain.

Questions for authors
  1. Can the authors report at least three independent evolution runs, including the distribution of final-round and selected-round performance, and use a held-out selection set so that model selection and final evaluation are separated?

  2. How sensitive is the HarnessX comparison to the three evaluator-authored seed harnesses? For GAIA, can the experiment be repeated in an environment that excludes benchmark mirrors and answer copies, or can the authors otherwise quantify the effect of this potential leakage?

5 comments

We appreciate the reviewer’s valuable comments and insightful questions, which we address point by point below.

Weakness 1 & Question 1

On more independent evolution runs

BenchmarkSettingMethodIn-loop peak pass@2 (1st run)In-loop peak pass@2 (2nd run)In-loop peak pass@2 (3rd run)Test pass@2 (1st run)Test pass@2 (2nd run)Test pass@2 (3rd run)Test pass@2 (Avg)API cost ($)HarnessX / LiteEvo
ALFWorldTransductiveHarnessX94.093.395.593.8 ± 2.289.6 ± 0.895.3 ± 0.492.9 ± 3.0141.45 ± 30.25--
LiteEvo96.394.293.496.3 ± 0.793.5 ± 0.995.3 ± 0.995.0 ± 1.48.16 ± 0.6717.3×
InductiveLiteEvo95.093.994.494.8 ± 0.792.5 ± 0.793.5 ± 0.493.6 ± 1.18.79 ± 1.6616.1×
WebShopTransductiveHarnessX66.068.068.063.0 ± 3.063.7 ± 0.663.3 ± 4.563.3 ± 0.3148.51 ± 29.66--
LiteEvo64.269.766.563.0 ± 0.068.3 ± 0.666.7 ± 0.666.0 ± 2.710.46 ± 1.4514.2×
InductiveLiteEvo52.347.252.370.0 ± 1.060.0 ± 3.060.3 ± 4.063.4 ± 5.710.30 ± 0.4714.4×
AppWorldTransductiveHarnessX63.163.761.962.3 ± 0.963.3 ± 3.861.3 ± 1.662.3 ± 1.0192.03 ± 21.51--
LiteEvo65.568.255.867.7 ± 0.963.9 ± 1.562.9 ± 2.164.8 ± 2.512.52 ± 1.7115.3×
InductiveLiteEvo63.259.964.965.5 ± 1.262.3 ± 0.365.1 ± 1.564.3 ± 1.713.17 ± 1.7514.6×
GAIATransductiveHarnessX68.068.970.964.4 ± 1.562.5 ± 4.663.4 ± 2.263.4 ± 1.0169.24 ± 11.47--
LiteEvo64.662.364.165.2 ± 1.662.1 ± 1.764.1 ± 1.763.8 ± 1.611.47 ± 0.7714.8×
SWE-bench VerifiedTransductiveHarnessX67.360.060.057.0 ± 6.457.0 ± 2.855.2 ± 5.856.4 ± 1.0176.95 ± 7.85--
LiteEvo60.063.672.261.8 ± 4.864.8 ± 4.263.0 ± 2.863.2 ± 1.526.40 ± 4.326.7×
InductiveLiteEvo67.366.767.369.1 ± 4.864.8 ± 1.064.8 ± 4.266.3 ± 2.421.37 ± 9.448.3×

To assess the reproducibility of our results, we conducted three independent evolution runs for each setting, with the results reported in the table above. For each run, we evaluated the selected round on the test set three times. Round selection was based on in-loop performance in the transductive setting and validation performance in the inductive setting.

In the transductive setting, LiteEvo achieved higher test pass@2 than HarnessX on all five benchmarks when averaged across the three runs. However, the margin on GAIA was very small, at 0.4 percentage points.

In the inductive setting, although LiteEvo does not access test tasks during evolution, it matched or exceeded transductive HarnessX on all four evaluated benchmarks, including a 0.1 percentage-point gain on WebShop. Thus, across both settings, LiteEvo matched or exceeded HarnessX’s performance while reducing API costs by factors of 6.7–17.3 across benchmarks and approximately 12 overall.

On final-round versus selected-round performance

BenchmarkSettingMethod1st run In-loop (Best)1st run In-loop (Final)1st run Test pass@2 (Best)1st run Test pass@2 (Final)2nd run In-loop (Best)2nd run In-loop (Final)2nd run Test pass@2 (Best)2nd run Test pass@2 (Final)3rd run In-loop (Best)3rd run In-loop (Final)3rd run Test pass@2 (Best)3rd run Test pass@2 (Final)
ALFWorldTransductiveHarnessX94.093.393.8 ± 2.296.3 ± 0.793.391.889.6 ± 0.892.8 ± 1.195.595.595.3 ± 0.495.3* ± 0.4
LiteEvo96.392.396.3 ± 0.790.5 ± 1.194.292.593.5 ± 0.993.0 ± 0.993.493.495.3 ± 0.995.3* ± 0.9
InductiveLiteEvo95.094.894.8 ± 0.790.8 ± 0.493.993.992.5 ± 0.792.5* ± 0.794.494.493.5 ± 0.493.5* ± 0.4
WebShopTransductiveHarnessX66.063.063.0 ± 3.062.0 ± 3.068.068.063.7 ± 0.663.7* ± 0.668.062.063.3 ± 4.562.0 ± 4.0
LiteEvo64.254.863.0 ± 0.056.3 ± 0.669.768.368.3 ± 0.666.0 ± 1.066.564.866.7 ± 0.665.0 ± 3.0
InductiveLiteEvo52.345.370.0 ± 1.062.0 ± 1.047.247.260.0 ± 3.060.0* ± 3.052.352.360.3 ± 4.060.3* ± 4.0
AppWorldTransductiveHarnessX63.159.562.3 ± 0.961.7 ± 1.263.757.163.3 ± 3.863.7 ± 1.261.961.961.3 ± 1.661.3* ± 1.6
LiteEvo65.564.467.7 ± 0.959.9 ± 3.068.260.363.9 ± 1.560.9 ± 2.755.855.862.9 ± 2.162.9* ± 2.1
InductiveLiteEvo63.263.265.5 ± 1.265.5* ± 1.259.959.662.3 ± 0.362.1 ± 2.864.964.965.1 ± 1.565.1* ± 1.5
GAIATransductiveHarnessX68.064.164.4 ± 1.561.5 ± 1.168.958.362.5 ± 4.659.9 ± 3.170.966.063.4 ± 2.265.0 ± 3.5
LiteEvo64.664.665.2 ± 1.665.2* ± 1.662.355.262.1 ± 1.758.9 ± 2.464.160.264.1 ± 1.761.8 ± 3.9
SWE-bench VerifiedTransductiveHarnessX67.356.457.0 ± 6.455.8 ± 2.860.052.757.0 ± 2.850.3 ± 2.860.054.555.2 ± 5.857.6 ± 4.6
LiteEvo60.054.561.8 ± 4.857.0 ± 3.863.652.764.8 ± 4.252.7 ± 3.672.261.863.0 ± 2.863.6 ± 1.8
InductiveLiteEvo67.363.669.1 ± 4.860.0 ± 3.166.758.264.8 ± 1.063.6 ± 1.867.361.864.8 ± 4.257.6 ± 2.1

To assess the impact of best-round selection on the results, we report both the selected-round and final-round performance for each evolution run in the table above. Runs in which the two rounds coincide are marked with an asterisk (*) in the final-round column.

Among the 18 LiteEvo runs in which the two rounds differ, the selected round achieved higher mean test performance than the final round in 17 runs. The only exception was the third transductive run on SWE-bench, where the difference was small, at 0.6 percentage points. In contrast, for HarnessX, the final round outperformed the selected round on the test set in 5 of the 12 runs in which the two rounds differ.

On held-out selection

BenchmarkDivisionTask poolTrain (evolve)Val (gate)Test (grade)Disjoint by
ALFWorldNativeALFRED game files134 of 3,553 (train)140 (valid-seen, all)134 (valid-unseen, all)Game
WebShopCarved6,910 goals, 415 distinct products100 (51 products)100 (58 products)100 (54 products)Product
AppWorldNativeReleased splits90 (train, all)57 (dev, all)168 (test-normal, all)Task
GAIAFixed subset103 text-only questionsTransductive only (the same 103)--
SWE-bench VerifiedCarved497 instances, 10 repos555555Instance

Held-out selection is already applied in LiteEvo’s inductive experiments. For all inductive runs, the train/validation/test splits are fixed in advance and are disjoint at the game, product, task, or instance level, as shown in the table above. The meta-agent evolves the harness based on trajectories generated on training tasks, while the gate uses separate rollouts on validation tasks to make accept/hold/revert decisions at each round. The resulting harness is then evaluated on test tasks that the evolution loop has never seen. Further details on evaluation independence are provided in our response to Reviewer pKt5 (“On independence of evaluation”).

Weakness 2 & Question 2

BenchmarkSeed harnessIn-loop pass@2: R0In-loop pass@2: PeakIn-loop pass@2: Peak roundIn-loop pass@2: R0 → peakTest pass@2
ALFWorldHarnessX-styled seed62.794.3 ± 1.1R6 / R5 / R7+31.692.9 ± 3.0
LiteEvo neutral seed53.794.0R4+40.390.8 ± 1.7
WebShopHarnessX-styled seed41.767.3 ± 1.2R6 / R7 / R4+25.763.3 ± 0.3
LiteEvo neutral seed12.065.0R4+53.066.0 ± 3.6
AppWorldHarnessX-styled seed47.462.9 ± 0.9R6 / R5 / R7+15.562.3 ± 1.0
LiteEvo neutral seed0.054.2R6+54.251.0 ± 3.6
SWE-bench VerifiedHarnessX official seed57.062.4 ± 4.2R1 / R2 / R3+5.556.4 ± 1.0
LiteEvo neutral seed50.956.4R3+5.553.3 ± 3.8

On the sensitivity of HarnessX to seed harness

Except for GAIA, whose official recipe is fully available, our HarnessX reproduction used reasonably constructed seed harnesses or evolution recipes. We therefore conducted one additional evolution run for each of the other four benchmarks using LiteEvo’s neutral initial harness as the seed, with three test evaluations per run. The results are shown in the table above. The evolution loop, meta-agent, runner, and turn/token budgets were kept identical to the original implementation. For a more stable comparison, we used statistics from the three independent evolution runs requested by the reviewer for the original HarnessX reproduction (with HarnessX-styled or HarnessX official seeds).

Except on AppWorld, the choice of seed harness did not lead to substantial differences in test pass@2. On all three benchmarks, the gap between the two arms falls within one standard deviation of one arm: the gaps on WebShop (2.7 percentage points) and SWE-bench Verified (3.1 percentage points) fall within the neutral arm’s standard deviations (±3.6 and ±3.8, respectively), while the gap on ALFWorld (2.1 percentage points) falls within the HarnessX arm’s standard deviation (±3.0). The direction of the difference is also inconsistent, with the neutral seed actually performing better on WebShop. On AppWorld, the neutral seed started with no knowledge of the required method calls for task termination. HarnessX’s meta-agent did not discover all of these requirements within eight rounds, resulting in lower performance.

These results suggest that the seeds we constructed did not disadvantage HarnessX. On ALFWorld and WebShop, the differences show no consistent direction, while on AppWorld, our seed actually benefited HarnessX (averaging 72.8 vs. 69.3 across the three benchmarks). Had both methods used the same neutral initial harness, the comparison would likely have been more favorable to LiteEvo.

Reviewer_CzboOpenReview ↗
Rating
3
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper automates the design of the harnes: the system prompt, small helper processors, and runner settings. LiteEvo starts every benchmark from the same short neutral harness, and four tool-free meta-agents, which are never told the benchmark's name, read failure trajectories, keep reusable parts in a library, and compose a candidate each round.

Critical strengths

The idea is well posed. The prior work uses hand-written starting harnesses and spends a large token budget, so testing a cheap blind search is important. The evaluation is careful. The train, val, and test splits are disjoint and hash-checked.

Critical weaknesses

The benchmark-blind claim and the numbers are hard to check. The meta-agent prompts contain priors measured on these same benchmarks, the code contains steps the paper does not describe, and no run logs are included, so the tables cannot be verified.

Questions for authors
  1. Do validation trajectories reach the meta-agents when they build a harness? Section 2 says only train trajectories may be read, but Section 3.4 suggests otherwise.

  2. Which all-fail group setting produced Table 1? The paper picks the three lowest groups, but the code default and its own help text point to different settings.

  3. How much does the 13x cost gap change if the baseline gets the same 15-round budget and the cost logs follow Appendix C.3 exactly?

  4. Why do unseen test tasks score higher than the evolution tasks on WebShop and SWE-bench? Is this split difficulty imbalance?

  5. Please fix small issues: the SICA workshop name, the Meta-Harness venue, and the 0.6 point figure that is missing from the cited paper.

6 comments

We thank the reviewer for the thoughtful feedback and constructive questions. We provide detailed responses to each point below.

Weakness

Priors in the meta-agent prompts

IDLocationRemoved measurement / wordingSource (benchmark / split)Type
A1CommonMedian fragment of 536 chars, median prompt of 3.6kDevelopment archive statisticsSplit-agnostic
A2CommonDeleting the worked examples cost 8.9 pp pass@2 (p=0.03); 42.8% of episodes reused an example lineAppWorld test-normal (168)Test
A3CommonMulti-line actions went from 2.0% to 25.5% of turnsAppWorld test-normal (57 tasks)Test
A4CommonGeneric introspection changed the score by +0.0 pp (22/22, p=1.00)AppWorld test-normal (168)Test
A5Common"All seven sat in the first third"AppWorld test-normal (same experiment as A2)Test
A6Common19 protocol components were written, and none was ever selectedDevelopment archive statisticsSplit-agnostic
A8CommonAn observation cap of 4,000 cost 13.6 pp (30.4 → 44.6)AppWorld test-normal (168)Test
A9a/bCommon"the mug is in cabinet 4", "go to cabinet 1", "a single appliance"ALFWorld vocabulary (not a measurement)Benchmark-like example
A11Common"page or scene" textDescribes the WebShop and ALFWorld formats (not a measurement)Benchmark-like example
A12Analyst"Track which containers you have already searched"ALFWorld-like exampleBenchmark-like example
A13Analyst1,585 components in 80 libraries; 30.0 pp of train pass@2; 42.8%Development archive and trainTrain
A15Composer90 of 90 episodes failed to act on turn 1; 19% of turns lostAppWorld train (90, in-loop)Train
A16Composer342 episodes, 2.0% → 25.5%AppWorld test-normal (same experiment as A3)Test

As the reviewer pointed out, we confirmed that parts of the meta-agent prompts contained priors measured on the same benchmarks. We therefore audited every prompt the meta-agents receive. For each sentence, we traced the development measurement or example it came from, and we removed or revised the sentences that carried such information (an additional blindness patch). The first table above lists examples of the removed sentences and their sources. They include numbers measured on AppWorld's test split, numbers measured on the train split, and example sentences that evoke the formats of ALFWorld or WebShop. The full revised instructions are in Appendix F. The submitted instructions are kept unchanged in Figures 4–12 for comparison.

BenchmarkSettingRevised prompts: In-loop peak pass@2Revised prompts: Test pass@2 (mean ± SD over three test runs; one evolution run)Before (avg. of three evolution runs): In-loop peak pass@2Before (Mean ± SD across three evolution runs, each evaluated three times: Test pass@2
ALFWorldTransductive95.092.3 ± 1.194.6 ± 1.595.0 ± 1.4
Inductive93.392.8 ± 1.194.4 ± 0.693.6 ± 1.1
WebShopTransductive70.769.3 ± 1.566.8 ± 2.866.0 ± 2.7
Inductive51.367.3 ± 2.150.6 ± 2.963.4 ± 5.7
AppWorldTransductive64.564.5 ± 1.263.2 ± 6.564.8 ± 2.5
Inductive60.263.1 ± 0.662.7 ± 2.564.3 ± 1.7
GAIATransductive63.361.8 ± 1.163.7 ± 1.263.8 ± 1.6
SWE-bench VerifiedTransductive69.162.4 ± 1.065.3 ± 6.363.2 ± 1.5
Inductive65.564.8 ± 2.167.1 ± 0.366.3 ± 2.4

The second table above shows the results of rerunning evolution in each setting with the revised prompts. Test pass@2 ranges from 2.7 pp lower to 3.9 pp higher than before the revision. The average over the nine settings is 70.9, nearly the same as the 71.2 average before the revision. This suggests that LiteEvo's performance does not depend much on the test-derived sentences the reviewer identified.

Steps in the code

We explain the inconsistencies between the submitted code and the reported runs, and how we fixed them in the revised supplementary material, in our answer to Question 2. If the reviewer points us to specific steps that the paper does not describe well enough, we will add them to the appendix.

Run logs

We have added LiteEvo.html to the revised supplementary material. It reproduces one complete round, round 6, of an AppWorld transductive evolution run. The file covers the rollouts (168 tasks × 4 samples), the actual prompt, input and output of each of the four meta-agents, the harness that the loop code assembles from their output, and the gate decision on that harness in the next round. Clicking any step opens the actual text that passed through it. The page is also available at the anonymized site https://anonymous.4open.science/w/liteevo-iclr2027/.

Reliability of the numbers

BenchmarkSettingMethodIn-loop peak pass@2 (1st run)In-loop peak pass@2 (2nd run)In-loop peak pass@2 (3rd run)Test pass@2 (1st run)Test pass@2 (2nd run)Test pass@2 (3rd run)Test pass@2 (Avg)
ALFWorldTransductiveHarnessX94.093.395.593.8 ± 2.289.6 ± 0.895.3 ± 0.492.9 ± 3.0
LiteEvo96.394.293.496.3 ± 0.793.5 ± 0.995.3 ± 0.995.0 ± 1.4
InductiveLiteEvo95.093.994.494.8 ± 0.792.5 ± 0.793.5 ± 0.493.6 ± 1.1
WebShopTransductiveHarnessX66.068.068.063.0 ± 3.063.7 ± 0.663.3 ± 4.563.3 ± 0.3
LiteEvo64.269.766.563.0 ± 0.068.3 ± 0.666.7 ± 0.666.0 ± 2.7
InductiveLiteEvo52.347.252.370.0 ± 1.060.0 ± 3.060.3 ± 4.063.4 ± 5.7
AppWorldTransductiveHarnessX63.163.761.962.3 ± 0.963.3 ± 3.861.3 ± 1.662.3 ± 1.0
LiteEvo65.568.255.867.7 ± 0.963.9 ± 1.562.9 ± 2.164.8 ± 2.5
InductiveLiteEvo63.259.964.965.5 ± 1.262.3 ± 0.365.1 ± 1.564.3 ± 1.7
GAIATransductiveHarnessX68.068.970.964.4 ± 1.562.5 ± 4.663.4 ± 2.263.4 ± 1.0
LiteEvo64.662.364.165.2 ± 1.662.1 ± 1.764.1 ± 1.763.8 ± 1.6
SWE-bench VerifiedTransductiveHarnessX67.360.060.057.0 ± 6.457.0 ± 2.855.2 ± 5.856.4 ± 1.0
LiteEvo60.063.672.261.8 ± 4.864.8 ± 4.263.0 ± 2.863.2 ± 1.5
InductiveLiteEvo67.366.767.369.1 ± 4.864.8 ± 1.064.8 ± 4.266.3 ± 2.4

To make the numbers more reliable, we ran three independent evolution runs for each benchmark and setting. We evaluated the harness returned by each run three times on the test set. We ran and evaluated HarnessX, the baseline, in the same way, with three evolution runs per benchmark. The table below gives the result of each run. It shows that the numbers do not hinge on any single run, and it shows the spread across runs directly.

Question 1

No, every trajectory a meta-agent reads comes from the train split. The meta-agents see the val split only as scores. The journal records each round's val pass@2 with the gate's decision and the spread between workers, and a note after a revert repeats the two pass@2 values the gate compared. No val episode or per-task val outcome enters any meta-agent's input.

The last sentences of Section 3.4 describe timing, and we will reword them. At the start of round t+1, the candidate's train and val rollouts run in one pass. The meta-agents then compose the next candidate from the train rollouts, while the val rollouts only yield the score s_t that the gate compares with b*. "Composed from them" was meant to refer to the train rollouts.

The supplementary code follows the same split._score_val in liteevo/loop.py writes val episodes to disk for auditing and returns only aggregate scores. The analyst's trajectory sample, the composer's excerpt of failed trajectories and the editor's 24-episode processor replay all come from the round's train rollouts.

We will revise Section 3.4 to: "The meta-agents compose the next candidate from the train rollouts before the gate decides, and they see the val rollouts only through the score s_t recorded in the journal." We will also extend item (ii) of Section 2 to "val tasks, used to choose between harnesses, whose trajectories no meta-agent reads."

Question 2

We apologize for the confusion. Every run in Table 1 used the setting described in the paper. The analyst's sample consists of the ten mixed groups with the highest mean partial credit and the three all-fail groups with the lowest mean partial credit. On SWE-bench Verified, as the caption of the settings table in the appendix states, the sample has five mixed groups and two all-fail groups, and the all-fail groups are again those with the lowest mean partial credit. Each run records its sampling settings in its history file. We checked these records for all 27 LiteEvo runs behind Table 1, and every run took its all-fail groups from the low end only.

The inconsistency you found came from the default value in the submitted code. The help text of --allfail-hi correctly stated that the reported runs used 0, the setting that takes all-fail groups only from the lowest mean partial credit. However, the code's default was still 1, a setting we tried during development, in which one of the three all-fail slots goes to the group that came closest to success. The reported runs were not affected by this default, because their launchers passed 0 explicitly. Still, since the default and the help text pointed to different settings, the uploaded code alone did not make clear which one the paper used.

We have fixed this in the revised supplementary material. --allfail-hinow defaults to 0, so by default the all-fail groups are those with the lowest mean partial credit, as in every reported run. The SWE-bench launchers now default to five mixed groups and two all-fail groups. The help texts, the launcher comments and docs/RUNBOOK.mdhave been updated to match these defaults.

Question 3

BenchmarkMethodBest round (15 rounds)In-loop peak (15 rounds)Test pass@2 (15 rounds)API cost ($), 8 roundsAPI cost ($), 15 roundsHarnessX / LiteEvo, 8 roundsHarnessX / LiteEvo, 15 rounds
ALFWorldHarnessXR1394.893.5 ± 1.9117.79219.2213.3×12.9×
LiteEvoR1595.095.0 ± 1.18.8517.02
WebShopHarnessXR1270.064.7 ± 1.5175.11441.0215.9×21.2×
LiteEvoR1269.869.0 ± 2.010.9820.85
AppWorldHarnessXR1367.362.3 ± 1.7229.94463.8318.3×20.0×
LiteEvoR966.666.3 ± 0.912.5423.21
GAIAHarnessXR568.962.5 ± 4.6180.03440.7217.0×19.2×
LiteEvoR1566.263.1 ± 1.010.5923.00
SWE-bench VerifiedHarnessXR1060.053.9 ± 3.8186.42363.356.1×6.4×
LiteEvoR1565.562.4 ± 2.830.6056.48

We ran both methods for 15 rounds on each benchmark, with costs counted exactly as in Appendix C.3. Only the meta-agent calls that produce a harness that is later evaluated are counted: R1 to R7 for 8 rounds, and R1 to R14 for 15 rounds. Every input token is billed at the full price, whether it was cached or not. As in Appendix C.3, HarnessX evolution steps that failed and were re-run are counted only once, by their completed attempt.

Summed over the five benchmarks, HarnessX costs 12.1× as much as LiteEvo at 8 rounds ($889.29 vs. $73.56) and 13.7× as much at 15 rounds ($1,928.14 vs. $140.56). The ratio grows because the two methods' costs scale differently as rounds are added. From 8 to 15 rounds, LiteEvo's cost grows by a factor of 1.85 to 2.17. This is close to the ratio of rounds (15/8 ≈ 1.9), so its cost per round stays roughly constant. HarnessX's cost grows by a factor of 1.86 to 2.52, reaching about 2.5 on WebShop and GAIA, so its cost per round rises as the run goes on. At 15 rounds, LiteEvo also still has a higher test pass@2 than HarnessX on all five benchmarks (71.2 vs. 67.4 on average).

Therefore, with the same 15-round budget, the cost gap does not shrink but grows, and LiteEvo keeps its accuracy advantage.

Reviewer_pKt5OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

LiteEvo proposes a lightweight procedure for optimizing the harness around a frozen language-model agent. Starting from a shared neutral configuration, it uses a small number of structured model calls to analyze execution trajectories, maintain a component library, and assemble improved prompts, processors, and runtime settings. The paper reports competitive performance at substantially lower meta-agent API cost than its HarnessX reproduction, together with gains on tasks held out from the optimization loop. Its central message is that effective harness adaptation can be inexpensive and need not rely on extensive agentic search.

My main reservation is that the paper offers limited insight into why this lightweight approach works and when it should be expected to work. Trajectory-driven harness improvement is already established, and performance gains from supplying missing interface knowledge, operational guidance, and examples are largely expected. Although the proposed organization has practical value, the experiments do not clearly establish how much it adds beyond these familiar effects or a simpler adaptation procedure.

This distinction matters for the paper’s broader message. The results demonstrate that lightweight optimization can work in the tested settings; but do they work in more realistic settings? for example, most of tested tasks like WebShop, AppWorld are simplified simulated environments. Much of the observed benefit may come from correcting local, recurring execution problems in relatively controlled environments.

Therefore, It remains unclear whether the same approach can address longer task dependencies, sparse feedback, or failures whose causes require tracing across many steps. A stronger contribution would identify the conditions that make lightweight optimization effective and test its boundaries. Without that explanation, I view LiteEvo as a useful engineering demonstration, with limited reusable or workable insights for a broadly applicable principle of efficient agent optimization.

Critical strengths

The low-cost results across several environments are useful.

Critical weaknesses
  1. The independence of the generalization evaluation is unresolved. LiteEvo's shared meta-agent instructions include AppWorld development measurements involving the split used for its inductive results (B.9). The SWE analyst's statistic counting episodes without file edits was also added after inspecting an earlier run's test-split trajectories (C.2). These disclosures show that evaluation feedback influenced method design, although overlap with the final test tasks remains unclear. Holding tasks out of the evolution loop alone therefore does not establish independent evaluation. The authors should clarify which tasks informed development and whether the final test remained untouched.

  2. The contribution offers limited insight beyond familiar adaptation effects. Trajectory-driven harness improvement is already established, and gains from supplying operational guidance and examples are largely expected. LiteEvo's inexpensive component-library workflow is useful, but its advantage over simpler adaptation remains insufficiently isolated. For example, a one-line format hint reaches 61.2 pass@2 on SWE-bench Verified, compared with 63.2 for the evolved transductive harness. The broader question is when lightweight optimization is sufficient: the current results do not explain whether small trajectory samples can diagnose failures involving long dependencies or sparse feedback. A comparison with a simple optimizer using the same feedback and budget, together with tests of these conditions, would establish more clearly what the proposed mechanism adds and where it applies.

  3. The comparative advantage is not yet robustly established. The HarnessX reproduction replaces its multi-agent optimizer with a single agent, disables extended thinking on four benchmarks, and uses incomplete ALFWorld seed coverage. These changes complicate interpretation of LiteEvo's 3.7-point average advantage, although not all baseline modifications favor LiteEvo. Each setting also has only one complete evolution run; the three final evaluations repeat the same returned harness. Comparable baseline configurations and independent evolution runs are needed to establish the stability of the advantage. The reported API savings remain useful, but exclude task-execution costs.

  4. The effective configuration is unclear because the method description and supplied prompts disagree. LiteEvo's default has one analyst, yet the editor prompt asks for agreement among several independent analysts. The composer instructions also disagree about who supplies reply formatting and whether component selection uses only an index or includes component bodies. These discrepancies may reflect outdated templates rather than execution errors, but leave uncertainty about which procedure produced the results. The authors should provide the actual prompts and one round's inputs, outputs, and assembled harness.

Questions for authors

see the weaknesses

9 comments

We thank the reviewer for the constructive feedback and thoughtful questions. We address each point below.

Weakness 1 & Weakness 4

On meta-agent instruction issues

The sentences the reviewer pointed out are wording used in the multiple-analyst per-worker setup (Table 15). This wording was not used in the main experiments, which use the single-analyst grouping setup, but by our mistake the previous version was included in the PDF. Appendix C.2 of the submission was wrong. The analyst in the SWE-bench Verified inductive run did not receive the share of episodes without a file edit. We added that statistic as a debugging option after inspecting test-split episodes of an earlier run, and it is off in all our SWE-bench Verified runs, transductive and inductive. The revision corrects C.2 (in blue). We are truly sorry for the confusion, and we highlighted the related parts in blue in Appendix Section F.

In addition, we provide a full per-round trajectory as a web page. On this web page, you can see all inputs and outputs of the agent and the meta-agents in a round, as well as the assembled harness. Clicking each box or arrow shows its text at the bottom of the page, so please check it there. The page is anonymized and is also included in the supplementary materials. URL: https://anonymous.4open.science/w/liteevo-iclr2027/

On independence of evaluation

BenchmarkDivisionTask poolTrain (evolve)Val (gate)Test (grade)Disjoint by
ALFWorldNativeALFRED game files134 of 3,553 (train)140 (valid-seen, all)134 (valid-unseen, all)Game
WebShopCarved6,910 goals, 415 distinct products100 (51 products)100 (58 products)100 (54 products)Product
AppWorldNativeReleased splits90 (train, all)57 (dev, all)168 (test-normal, all)Task
GAIAFixed subset103 text-only questionsTransductive only (the same 103)--
SWE-bench VerifiedCarved497 instances, 10 repos555555Instance

The table above summarizes the train, val and test sets used for harness evolution on each benchmark. The transductive mode follows the same protocol as HarnessX: it evolves, gates and grades while looking directly at the test set. Therefore, we do not claim that the transductive results are an independent evaluation. The inductive mode evolves on train, gates on val, and then grades on test. The three splits are fixed lists of tasks and do not overlap at the level of games, products, tasks or instances.

IDLocationRemoved measurement / wordingSource (benchmark / split)Type
A1CommonMedian fragment of 536 chars, median prompt of 3.6kDevelopment archive statisticsSplit-agnostic
A2CommonDeleting the worked examples cost 8.9 pp pass@2 (p=0.03); 42.8% of episodes reused an example lineAppWorld test-normal (168)Test
A3CommonMulti-line actions went from 2.0% to 25.5% of turnsAppWorld dev (57 tasks)Test
A4CommonGeneric introspection changed the score by +0.0 pp (22/22, p=1.00)AppWorld test-normal (168)Test
A5Common"All seven sat in the first third"AppWorld test-normalTest
A6Common19 protocol components were written, and none was ever selectedDevelopment archive statisticsSplit-agnostic
A8CommonAn observation cap of 4,000 cost 13.6 pp (30.4 → 44.6)AppWorld test-normal (168)Test
A9a/bCommon"the mug is in cabinet 4", "go to cabinet 1", "a single appliance"ALFWorld vocabularyBenchmark-like example
A11Common"page or scene" textDescribes the WebShop and ALFWorld formatsBenchmark-like example
A12Analyst"Track which containers you have already searched"ALFWorld-like exampleBenchmark-like example
A13Analyst1,585 components in 80 libraries; 30.0 pp of train pass@2; 42.8%Development archive and trainTrain
A15Composer90 of 90 episodes failed to act on turn 1; 19% of turns lostAppWorld train (90, in-loop)Train
A16Composer342 episodes, 2.0% → 25.5%AppWorld devTest

As the reviewer pointed out, leaving the test tasks out of the loop does not by itself establish an independent evaluation. So we additionally checked the entire prompt given to the meta-agents, traced where each piece of information from the development process came from, and revised it (Additional blindness patch). The table above shows examples of the sources among the applied changes. We removed or revised all of this information, and the corresponding parts are highlighted in red in Appendix Section F.

BenchmarkSettingRevised prompts: In-loop peak pass@2Revised prompts: Test pass@2 (mean ± SD over three test runs; one evolution run)Before (avg. of three evolution runs): In-loop peak pass@2Before (mean ± SD across three evolution runs, each evaluated three times): Test pass@2
ALFWorldTransductive95.092.3 ± 1.194.6 ± 1.595.0 ± 1.4
Inductive93.392.8 ± 1.194.4 ± 0.693.6 ± 1.1
WebShopTransductive70.769.3 ± 1.566.8 ± 2.866.0 ± 2.7
Inductive51.367.3 ± 2.150.6 ± 2.963.4 ± 5.7
AppWorldTransductive64.564.5 ± 1.263.2 ± 6.564.8 ± 2.5
Inductive60.263.1 ± 0.662.7 ± 2.564.3 ± 1.7
GAIATransductive63.361.8 ± 1.163.7 ± 1.263.8 ± 1.6
SWE-bench VerifiedTransductive69.162.4 ± 1.065.3 ± 6.363.2 ± 1.5
Inductive65.564.8 ± 2.167.1 ± 0.366.3 ± 2.4

Above are the results of LiteEvo with the meta-agent prompts after the Additional blindness patch. Even after removing these sentences, performance stayed at a similar level overall. This suggests that LiteEvo's performance does not depend much on the test-derived instruction sentences that were pointed out.

Weakness 2

BenchmarkSettingLiteEvo (avg. of three runs): In-loop peak pass@2LiteEvo (avg. of three runs): Test pass@2Unified meta-agent: In-loop peak pass@2Unified meta-agent: Test pass@2
ALFWorldTransductive94.6 ± 1.595.0 ± 1.498.498.5 ± 0.7
Inductive94.4 ± 0.693.6 ± 1.193.592.3 ± 0.9
WebShopTransductive66.8 ± 2.866.0 ± 2.759.254.7 ± 1.5
Inductive50.6 ± 2.963.4 ± 5.759.069.7 ± 0.6
AppWorldTransductive63.2 ± 6.564.8 ± 2.510.39.7 ± 0.3
Inductive62.7 ± 2.564.3 ± 1.720.233.5 ± 2.9
GAIATransductive63.7 ± 1.263.8 ± 1.660.756.0 ± 0.6
SWE-bench VerifiedTransductive65.3 ± 6.363.2 ± 1.561.861.8 ± 1.8
Inductive67.1 ± 0.366.3 ± 2.465.563.6 ± 3.1

Unified meta-agent experiment

As the reviewer notes, trajectory-driven harness evolution is an established idea; LiteEvo's aim is to make it cost-efficient while keeping it fully automated. To show what LiteEvo's four role-specific meta-agents add, we implemented a single unified meta-agent. Under the main protocol, we ran one harness evolution loop per benchmark and setting and graded each returned harness three times. The table above compares it with the mean of LiteEvo's three runs in the revised Table 1. The unified meta-agent receives the agent's trajectories with a single system prompt adapted from LiteEvo's per-role prompts and outputs the assembled harness directly. It therefore skips the analyst's proposals, the editor's and librarian's selection and curation, and the composer's selection and assembly, and keeps no library or journal across rounds. It reads the same failure-oriented sample as LiteEvo's analyst, 13 task groups per round (7 on SWE-bench Verified), and keeps the same rounds, pass@2 gate and choice of the returned harness. It makes only 4 to 7 Opus 4.6 calls per run, against 40 to 43 for LiteEvo, and places no cap on the output length of each call. The API cost of the two is about the same:12.60 USD per run on average for the unified meta-agent against 13.63 USD for LiteEvo.

Comments on the paper

1 comment

Dear Reviewers,

During this rebuttal period, the draft and the supplementary materials can be revised, so we are leaving this common comment to inform all reviewers of the changes. In the draft, the revised parts are written in colored text so they can be told apart. Once again, we apologize to the reviewers for the confusion.

[Both the draft PDF and supplementary materials have been updated as of November 14, 2026, at 4:30 p.m. AoE.]

Draft

  • Table 1 has been revised. Previously, it consisted of only three evaluations of a single evolution run. We updated it with the results of three separate evolution runs, each evaluated three times, for nine evaluations in total.
  • Appendix Section F, "Updated Meta-agents' Instructions", has been added. We corrected the outdated version that had been included (blue-colored font), and added the prompts with an additional blindness patch for a more rigorous evaluation of independence (red-colored font).

Supplementary Materials

  • Of the draft changes above, the second one is applied: the outdated prompts have been updated. The additional blindness patch is not applied.
  • We fixed a wrongly set default of --allfail-hi, which sets how many of the all-failed task groups shown to the analyst come from the near-miss end. As its help text already stated, the default is 0: the analyst sees the three all-failed task groups with the lowest mean frac (two on SWE-bench Verified), as in every reported run.
  • We added LiteEvo.html. This web page visualizes the full process of round 6 of LiteEvo. The page is also available at the anonymized site https://anonymous.4open.science/w/liteevo-iclr2027/.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.