acceptodds
Under review as a conference paper at ICLR 2027

ParetoSaddler: Pareto-Guided Optimization of Agent Harnesses for Quality and Cost

Abstract

Harness optimization can substantially improve LLM agents without modifying model weights, but existing methods primarily optimize response quality. Extending them to joint quality–cost optimization introduces a challenge: applying the deployment preference during intermediate updates can prematurely reject candidates that are valuable for later improvements. We introduce ParetoSaddler, which addresses this mismatch by separating intermediate candidate promotion from final deployment selection. At each iteration, an optimizer updates a parent harness to produce a candidate, and then compares the two over repeated executions and uses Bayesian bootstrap Pareto support to promote candidates showing credible progress in either quality or cost. Promoted candidates receive broader development evaluation and remain available as parents for later optimization steps. We evaluate ParetoSaddler on ClawMark and GAIA2 using domain-disjoint splits. On ClawMark, ParetoSaddler improves quality from 70.86% to 81.35% while reducing cost from 1.01 per task; on GAIA2, it improves quality from 56.22% to 62.89% while reducing cost from 0.45 per task. The Pareto-selected harnesses also achieve higher quality at lower cost than quality-only harness optimization. Analysis shows that Pareto-aware optimization retains intermediate candidates that would be discarded by deployment preference but can support later improvements, while Bayesian bootstrap screening filters out candidates whose gains are weakly supported across runs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, harness optimization, multi-objective optimization, self-improving agent

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_b7ZVOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Building on AutoSaddler, this paper optimizes an agent harness (the prompts, tools, and control logic around a fixed LLM) for both task quality and token cost. In each iteration, an optimizer LLM patches a parent harness, and both versions are run three times on a small batch of training tasks. The patch is promoted if at least 70% of Bayesian bootstrap draws show higher quality, or lower cost without a large quality drop. A promoted version is evaluated on the full development set and can serve as a parent later. The final preference, such as a cost limit, is applied only at the end, because a patch that looks worse under it may still lead to better versions. On ClawMark and GAIA2, the method reaches higher test quality and lower cost than the initial harness, and does better than the baselines. Ablations examine the promotion rule, repeated runs, and Bayesian screening.

Critical strengths

Strength 1: The method gives clear gains on two benchmarks, measured on unseen task groups

In Table 1, quality on ClawMark rises from 70.86% to 81.35%, and cost per task falls from USD 1.17 to USD 1.01. On GAIA2, quality rises from 56.22% to 62.89%, and cost falls from USD 0.78 to USD 0.45. The selected harnesses also beat the quality-only AutoSaddler baseline on both quality and cost. Because the data are split by scenario (ClawMark) or Universe (GAIA2), these gains are measured on task groups the optimizer never saw, though still within the same benchmark.

Strength 2: The paper shows a case where a quality-only gate fails, and ablates its key design choices

When the parent already solves every screening task, quality cannot go up, so a quality-only gate rejects every patch. In two GAIA2 examples in Appendix C (I-9 and I-17), screening quality stays at 100% while cost drops by about 40%, and both versions also score higher on the development set. So when quality is saturated, cost can still pick out useful patches. The paper also ablates the design: Figure 3 swaps the promotion rule, and Table 3 replaces Bayesian screening with a mean-based gate.

Critical weaknesses

Weakness 1: The gains from Pareto promotion are not separated from the extra evaluation budget

The paper's central claim is that the final preference should not be used to filter candidates during optimization. Two comparisons test this directly, but both also change the evaluation budget.

The first is Figure 3. The Pareto gate promotes 11 candidates and the MinCost gate only 3. Each promotion adds 60 task runs on the development set, while one round of local screening costs 18. Following the accounting in Table 6, the two runs use about 1,020 and 540 task runs, nearly a factor of two, and the Pareto run also has more parents to build on. Yet the test gap is modest: 75.16% vs 73.95% quality, and USD 0.82 vs USD 0.90 cost.

The second is the headline comparison: ParetoSaddler's highest-quality harness beats the quality-only AutoSaddler baseline by 7.49 points on ClawMark and 5.56 on GAIA2. Besides the gate, the two also differ in whether patch generation uses cost information (Appendix A). The number of promotions likely differs too. The Pareto gate takes the union of "quality goes up" and "cost goes down", with no limit on cost increase (ε_C = +∞), so even a patch with no real effect usually passes: its support is about 1 − ½ × ½ = 0.75 when quality and cost are unrelated, and close to 1 when they rise together. At the same γ = 0.7, a quality-only gate gives such a patch about 0.5 and rejects it, and it rejects every patch when all screening tasks are already solved. In the full ClawMark log, 12 of 17 candidates were promoted (Appendix I). But Table 6 only gives an upper bound that assumes every candidate is promoted. The actual number of runs is not reported, and neither is the number of promotions for the quality-only baseline.

So the results show that the full method works, but not yet that the gain comes from Pareto promotion itself rather than from promoting more candidates and spending more evaluation. The I-11 example has the same gap: it shows that a more expensive patch came before the gains at I-14 and I-16, not that those gains needed it. A control with a similar number of promotions or the same budget would answer this, for example a quality-only or cost-only gate with a lower γ that promotes about as many candidates.

Weakness 2: The paper does not say how many tasks it takes to pay back the cost of optimization

Lower cost is one of the paper's main goals, but the cost in Table 1 only counts per-task execution of the final harness, not the cost of finding it. This cost is not small. On GAIA2, 20 of 25 candidates were promoted (Appendix J), which by the accounting in Table 6 means about 4,500 task runs. At USD 0.45 to USD 0.78 per run, this is roughly USD 2,000 to USD 3,500, plus about USD 240 in optimizer cost (Table 11). The final harness saves USD 0.33 per task over the initial one, so by this rough estimate it takes about 7,000 to 11,000 tasks to recover the cost. Compared with the quality-only AutoSaddler, ParetoSaddler also spends more on the optimizer per patch (USD 9.62 vs USD 6.01) and likely promotes more candidates (Weakness 1). This does not mean the method is not worth it. The quality gain has value on its own, and with enough tasks the cost is recovered. But for a paper whose main goals include cost, the full optimization cost and the break-even task count should be reported, together with the deployment scale the method targets.

Questions for authors
  1. (Related to Weakness 1) How many candidates were promoted, and how many task runs were actually used, in the quality-only AutoSaddler baseline, in the two runs of Figure 3, and in the main ParetoSaddler runs on both benchmarks?

  2. (Related to Weakness 1) Is there a control with a similar number of promotions or the same budget? For example, if the quality-only gate used a lower γ and promoted as many candidates, would it catch up with ParetoSaddler?

  3. (Related to Weakness 2) Could you report the full optimization cost of each method on both benchmarks, including optimizer cost and the cost of task runs for screening and development evaluation? Relative to the initial harness and the quality-only AutoSaddler, how many later tasks are needed to recover it?

4 comments
AuthorsOpenReview ↗

Response to Reviewer b7ZV (1/2)

We thank the reviewer for the encouraging assessment and detailed feedback. We have conducted additional controlled experiments and provided comprehensive optimization-cost accounting to address the concerns about promotion frequency and evaluation expenditure.

Q1/W1: How many candidates were promoted, and how many task runs were actually used, in the quality-only AutoSaddler baseline, in the two runs of Figure 3, and in the main ParetoSaddler runs on both benchmarks?

We thank the reviewer for this question. The table below summarizes candidate promotions and task-execution counts across all six runs. We will include these details in the revised manuscript.

Experiment / methodProposalsPromotedTrain task runsCandidate Dev task runsInitial Dev task runsTotal task runs
ClawMark: AS-Quality171030060060960
ClawMark: PS-Pareto1712300720601,080
GAIA2: AS-Quality25184503,5101954,155
GAIA2: PS-Pareto25204503,9001954,545
ClawMark Figure 3: MinCost local gate17330018060540
ClawMark Figure 3: Pareto local gate1711300660601,020

In the main comparisons, PS-Pareto promotes only two more candidates than AS-Quality on each benchmark (12 vs. 10 on ClawMark; 20 vs. 18 on GAIA2), yet achieves substantially better held-out quality–cost trade-offs. In the Figure 3 ablation, the Pareto gate promotes 11 candidates versus 3 for MinCost, resulting in 1,020 versus 540 total task runs. To disentangle the effects of promotion strategy and evaluation budget, we further conducted promotion-matched and budget-matched controlled experiments, presented in our response to Q2/W1 below.

Q2/W1: Is there a control with a similar number of promotions or the same budget? For example, if the quality-only gate used a lower γ and promoted as many candidates, would it catch up with ParetoSaddler?

We thank the reviewer for this valuable suggestion. To isolate the benefits of Pareto-aware promotion from increased promotion frequency or optimization budget, we conducted two additional controlled experiments and analyzed promotion-matched results from our existing runs.

Controlled experiments. We choose the ClawMark MinCost setting (Figure 3) as an example, where the Pareto and MinCost gates differ substantially in promotion frequency (11 vs. 3). We introduce two controls: (1) a random promotion gate that matches the exact number of promotions (also suggested by Reviewer CjhS), and (2) an extended MinCost run that matches the total optimization budget (we found it challenging to guarantee matched promotions via lowering the threshold ). Specifically:

  • Promotion-matched random gate: Randomly promotes exactly 11 candidates across 17 proposals, matching the number of promoted candidates in Pareto gate.

  • Budget-matched MinCost gate: Extends MinCost optimization from 17 to 25 iterations, closely matching the Pareto gate's total optimization expenditure (\1,296.99vs.1,296.99 vs. \\1,290.85).

    The held-out ClawMark results are:

Local promotion ruleQuality (%) ↑Cost ($) ↓
Pareto gate75.160.82
MinCost gate73.950.90
Promotion-matched random gate70.381.12
Budget-matched MinCost gate70.690.99

The Pareto gate outperforms both controls on quality and cost: by 4.78 percentage points and \0.30/taskoverrandompromotion,andby4.47pointsand0.30/task over random promotion, and by 4.47 points and \\0.17/task over budget-matched MinCost. Notably, extending MinCost optimization does not guarantee better generalization: its additional search selects a candidate with improved development performance but worse held-out results. These findings suggest that neither additional promotions nor greater optimization expenditure alone explains the Pareto gate's advantage.

Complementary evidence from PS-Pareto vs. AS-Quality. Across both benchmarks, PS-Pareto promotes only two more candidates than AS-Quality (12 vs. 10 on ClawMark; 20 vs. 18 on GAIA2). Even at matched promotion counts (10 and 18, respectively), PS-Pareto achieves better held-out quality–cost trade-offs: 79.60%/0.92vs.73.86%/1.24 on ClawMark, and 62.89%/0.45vs.57.33%/0.61 on GAIA2.

Together, these results provide consistent evidence that ParetoSaddler's improvements are not simply attributable to additional promotions or greater optimization expenditure. We will incorporate these additional results into the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer b7ZV (2/2)

Q3/W2: Could you report the full optimization cost of each method on both benchmarks, including optimizer cost and the cost of task runs for screening and development evaluation? Relative to the initial harness and the quality-only AutoSaddler, how many later tasks are needed to recover it?

We thank the reviewer for highlighting the importance of optimization expenditure. We now provide a detailed accounting of optimization expenditure across both benchmarks, including optimizer, training, and development evaluation costs, along with estimates of deployment amortization.

ClawMark optimization costs ($)

MethodOptimizer costTrain costDev costTotal optimization cost
AS-Quality168.61484.52960.191,613.32
MH-MinCost41.912608.78-2,650.69
MH-Pareto31.382725.81-2,757.19
PS-MinCost191.26334.95770.791,297.00
PS-Pareto213.07332.95808.301,354.32

GAIA2 optimization costs ($)

MethodOptimizer costTrain costDev costTotal optimization cost
AS-Quality150.24313.002,649.953,113.19
MH-MinCost53.734279.24-4,332.97
MH-Pareto81.254945.55-5,026.80
PS-MinCost200.62211.871390.891803.38
PS-Pareto240.51214.391,903.762,358.66

Key finding: ParetoSaddler achieves stronger held-out quality–cost performance at lower total optimization expenditure. Despite higher optimizer-side costs, PS-Pareto reduces total expenditure relative to AS-Quality by 16.1% on ClawMark (\1,354vs.1,354 vs. \\1,613) and 24.2% on GAIA2 (\2,359vs.2,359 vs. \\3,113). Although ParetoSaddler incurs higher optimizer costs, the increasingly cost-efficient harnesses discovered during optimization reduce subsequent training and development evaluation expenses, particularly in later stages. This demonstrates that optimizing harness execution cost can also improve the efficiency of the optimization process itself.

We further estimate the number of deployed tasks required to recover the optimization expenditure through execution-cost savings relative to the initial harness.

BenchmarkMethodOptimization cost ($)Deployment cost/task ($)Number of Break-even tasks vs. Initial
ClawMarkPS-Pareto$^Q$1354.321.018465
ClawMarkPS-Pareto$^C$1354.320.925418
GAIA2PS-Pareto$^Q$2,358.660.457148
GAIA2PS-Pareto$^C$2,358.660.477609

Across both benchmarks, the optimization expenditure can be amortized after approximately 5.4k–8.5k deployed tasks. For illustration, a shared optimized harness serving 1,000 active users completing five tasks daily would reach break-even in approximately 1–2 days. While actual amortization depends on deployment scale and workload, these estimates suggest that the upfront optimization expenditure can be recovered rapidly in sufficiently high-volume deployments.

We will incorporate these cost breakdowns, amortization estimates, and a visualization of held-out quality–cost performance versus total optimization expenditure into the revised manuscript.


Thank you again for your valuable feedback. We welcome any further questions and discussion.

Thanks for your response. It addresses all my concerns. I think improving the efficiency to use token is as important as improving the ability to solve tasks. I trust this paper is worth to be accepted. I have raised my score from 3 to 4.

Dear Reviewer b7ZV,

Thank you very much for your encouraging feedback and for reconsidering your assessment. We are delighted that the additional experiments and analyses have addressed your concerns. We sincerely appreciate your constructive suggestions, which have helped us strengthen the paper.

Best regards, Authors

Reviewer_Nm8AOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies joint quality–cost optimization of LLM agent harnesses with fixed model weights. ParetoSaddler separates intermediate candidate promotion from final deployment selection: a candidate can advance when repeated executions provide sufficient Bayesian bootstrap support for improvement in either quality or cost, with a configurable allowance for regression in the other objective. Promoted candidates receive development-set evaluation and remain available for subsequent optimization. The framework is instantiated on AutoSaddler and evaluated on ClawMark and GAIA2. The reported development-selected harnesses improve test quality while reducing per-task token cost.

Critical strengths
  1. The reported improvements are remarkable. The quality-oriented Pareto endpoint improves ClawMark quality from 70.86% to 81.35% while reducing cost from 1.17 to 11.01 per task, and improves GAIA2 quality from 56.22% to 62.89% while reducing cost from 0.78 to 0.45.
  2. The local-gate ablation and trajectory analysis provide evidence that requiring immediate cost reduction can discard updates useful for subsequent refinement. This supports allowing intermediate trade-offs rather than enforcing the final preference at every step.
Critical weaknesses
  1. Experiments with finite cost-increase allowances would help clarify how the current ϵC=+∞\epsilon_C=+\infty setting affects final quality, cost, and search expenditure.

  2. With only three executions per task, promotion decisions may be sensitive to individual runs. Additional executions for candidates near the threshold would help assess their stability.

Questions for authors
  1. How do finite cost-increase allowances affect final results and search expenditure?

  2. Do promotion decisions for candidates near the threshold remain consistent when more executions are collected?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Nm8A

We thank the reviewer for the positive assessment and constructive suggestions. We have conducted additional analyses and experiments to further examine the robustness of our design choices.

Q1/W1: How do finite cost-increase allowances affect final results and search expenditure?

We thank the reviewer for this suggestion. We intentionally leave the cost-increase allowance unbounded because a substantial local cost increase may still reflect genuine task progress, and a finite cap risks prematurely rejecting valuable intermediate candidates. To examine this trade-off, we conducted a retrospective sensitivity analysis of both cost-increase and quality-loss allowances.

Cost-increase sensitivity. We retrospectively replayed the Bayesian promotion gate on archived parent–candidate evaluations from ClawMark and GAIA2, varying the cost-increase allowance (ϵC\epsilon_C) while keeping ϵQ=0.1\epsilon_Q=0.1 and γ=0.7\gamma=0.7 fixed.

Allowed cost increase (ϵC\epsilon_C)ClawMarkGAIA2
0%6/1711/25
5%7/1711/25
10%9/1711/25
20%11/1716/25
Unbounded12/1720/25

With a 20% cap, promotions decrease from 12 to 11 on ClawMark and from 20 to 16 on GAIA2, avoiding 60 and 780 Dev task executions, respectively. However, this saving comes with a potential decrease in solution quality: On ClawMark, all Dev Pareto solutions remain reachable under the 20% cap. On GAIA2, however, the original search paths leading to these solutions are disrupted. For example, I-23 passes its own screening criterion but becomes unreachable along its original lineage because its parent I-22 is rejected. Among the remaining reachable archived GAIA2 candidates, the highest observed Dev quality decreases from 61.54% to 58.97%, while the lowest cost increases from \0.36to0.36 to \\0.40 per task.

Quality-loss sensitivity. We also removed the quality-loss bound while keeping ϵC=+∞\epsilon_C=+\infty and γ=0.7\gamma=0.7. This admits only one additional candidate on each benchmark, suggesting a relatively limited effect on promotion frequency in the recorded searches.

Overall, these findings support our asymmetric allowance design: finite cost-increase caps can reduce evaluation expenditure but risk pruning optimization paths toward stronger quality–cost solutions, whereas the quality-loss bound has a comparatively limited effect on promotion decisions.

Q2/W2: Do promotion decisions for candidates near the threshold remain consistent when more executions are collected?

We thank the reviewer for pointing this out. We re-evaluated all 25 GAIA2 parent–candidate comparisons using R=3, 6, and 9 executions per task, keeping the candidate harnesses, screening tasks, and promotion threshold (γ=0.7\gamma = 0.7) fixed.

Overall stability. Decision agreement was 92% (23/25) between R=3 and R=6, 88% (22/25) between R=3 and R=9, and 96% (24/25) between R=6 and R=9. Notably, all 20 candidates promoted at R=3 remained promoted with more executions. All three changes from R=3 to R=9 were rejection-to-promotion.

Near-threshold stability. Five candidates had initial Bayesian promotion support within (±0.15\pm 0.15) of (γ=0.7\gamma=0.7). Their results are summarized below:

CandidateR = 3R = 6R = 9Decision (3 → 6 → 9)
I-30.640.411.00R → R → P
I-130.800.880.93P → P → P
I-60.590.280.36R → R → R
I-120.811.001.00P → P → P
I-80.840.990.98P → P → P
P = Promote; R = Reject. Candidates are ordered by proximity to γ=0.7\gamma = 0.7 at R = 3.

Four of five near-threshold decisions remained unchanged at R=9, including all three initial promotions. Only I-3 switched from rejection to promotion, suggesting that borderline rejections may benefit from additional evidence.

Overall, these results support R=3 as a practical, cost-efficient screening choice: it preserves all initial promotions and achieves 88% agreement with R=9, while requiring only one-third as many screening executions.


Thank you again for your valuable feedback. We welcome any further questions and discussion.

Thank you for the response and additional experiments. These results address my main concerns. Considering the score is already positive, I will maintain it.

Dear Reviewer Nm8A,

Thank you very much for your positive feedback. We are glad that our response and additional experiments have addressed your main concerns. We sincerely appreciate your constructive feedback and suggestions.

Best regards, Authors

Reviewer_CjhSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at automatic harness optimization when you care about both quality and cost. The key observation is that if you apply the final deployment preference at every intermediate step, you can throw away useful updates. A typical example is a patch that fixes premature termination: quality goes up, but so does cost. ParetoSaddler, built on AutoSaddler, promotes a candidate whenever there is credible progress in either objective, judged by a Bayesian bootstrap over repeated runs. The deployment preference is applied only at the end. On ClawMark and GAIA2, with domain-disjoint splits, it finds harnesses that are both better and cheaper than the starting point, and even beats quality-only optimization on quality.

Critical strengths

The motivation is clear and believable, especially the point that a harness can look cheap simply because it gives up early. Separating “worth exploring” from “worth deploying” is a clean way to frame the problem. I also like that each component gets an end-to-end ablation rather than just retrospective analysis. The case studies where mini-batch quality is already saturated, so cost is the only signal left, are intuitive. The paper is also transparent about patches and compute.

Critical weaknesses
  1. The main comparison mixes up “Pareto reasoning” with “being more permissive.” With no cap on cost increases, the Pareto gate accepts everything the quality gate accepts, plus cost-reducing candidates. Meanwhile, the quality-only gate requires a strict quality gain, so it can never promote anything on a mini-batch already at full score. AutoSaddler may lose mostly because it promotes fewer candidates and gets stuck on saturated batches. Since “adding cost improves quality” is the paper’s most interesting claim, I’d want controls: a non-strict quality gate, or random promotion at the same rate. The MinCost-gate ablation has the same issue, with far fewer promotions under the alternative gate.
  2. Most of the evidence comes from single optimization runs. Each ablation arm is one run, evaluated on small test sets, and the whole pipeline is stochastic (LLM optimizer, mini-batch sampling, agent rollouts). The extra GAIA2 run is reassuring but limited. Differences of a few points between arms could fall within run-to-run variation of the full pipeline. The ablations need repeats, or the claims should be softened.
  3. The Bayesian screening does less than the framing suggests. With three runs on each of three tasks, the bootstrap is reweighting a handful of numbers. It also ignores task sampling, which is probably the bigger source of noise with only three tasks. And the threshold was justified by how many candidates it promotes, not by downstream outcomes. The I-12 example is really about one task collapsing while the others don’t. A simple rule like “no single task regresses by more than some margin” might catch it equally well. A comparison against a couple of simple robust rules would show whether the Bayesian machinery is needed.
  4. The optimization edits environment code, not just the harness. On GAIA2, the final patches change the simulated apps themselves (email, contacts, calendar, shopping, cab), adding batched lookups, filters, and compact outputs. On ClawMark, they compact tool outputs and change read tools. Much of the cost saving likely comes from giving the agent better tools. That’s legitimate if you own the tools, but it isn’t the same as optimizing a harness against a fixed environment, and domain-disjoint splits don’t help because the apps are shared across domains. Please state which edits are allowed and add an ablation that restricts edits to prompts and agent logic. Some patches also look tailored to benchmark quirks.
  5. The method’s own cost deserves a fuller accounting. Optimizer cost per patch is noticeably higher than for the baselines, the reported execution budgets are upper bounds, and the single-run versus three-run comparison isn’t compute-matched. A plot of final test quality and cost against total optimization spend for each method would help, along with a rough sense of how many deployed tasks it takes for the savings to pay back the search.
Questions for authors

Does a non-strict quality gate, or random promotion at a matched rate, close the gap to ParetoSaddler? How much of the gain remains if patches can’t touch app or tool code? How stable are the ablation conclusions across repeated runs?

3 comments
AuthorsOpenReview ↗

Response to Reviewer CjhS (1/3)

We thank the reviewer for the thoughtful and constructive feedback, and for recognizing the motivation and empirical contributions of our work. We have conducted additional controlled experiments and analyses to address the concerns and clarify the scope of our approach.

Q1/W1: Does a non-strict quality gate, or random promotion at a matched rate, close the gap to ParetoSaddler?

We thank the reviewer for this insightful question. To disentangle the benefits of Pareto-aware promotion from those of increased promotion frequency or optimization expenditure, we conducted two additional controlled experiments, supplemented by an analysis of our existing runs.

Controlled experiments. We choose the ClawMark MinCost setting (Figure 3) as an example, where the Pareto and MinCost gates differ substantially in promotion frequency (11 vs. 3). We introduce two controls:

  • Promotion-matched random gate: As suggested, we randomly promote exactly 11 candidates across 17 proposals, matching the number of promoted candidates in Pareto gate.

  • Budget-matched MinCost gate: Extends MinCost optimization from 17 to 25 iterations, closely matching the Pareto gate's total optimization expenditure (\1,296.99vs.1,296.99 vs. \\1,290.85).

The held-out ClawMark results are:

Local promotion ruleQuality (%) ↑Cost ($) ↓
Pareto gate75.160.82
MinCost gate73.950.90
Promotion-matched random gate70.381.12
Budget-matched MinCost gate70.690.99

The Pareto gate outperforms both controls on quality and cost: by 4.78 percentage points and \0.30/taskoverrandompromotion,andby4.47pointsand0.30/task over random promotion, and by 4.47 points and \\0.17/task over budget-matched MinCost. Notably, extending MinCost optimization does not guarantee better generalization: its additional search selects a candidate with improved development performance but worse held-out results. These findings suggest that neither additional promotions nor greater optimization expenditure alone explains the Pareto gate's advantage.

Complementary evidence from PS-Pareto vs. AS-Quality. Across both benchmarks, PS-Pareto promotes only two more candidates than AS-Quality (12 vs. 10 on ClawMark; 20 vs. 18 on GAIA2). Also, at matched promotion counts (10 and 18, respectively), PS-Pareto achieves better held-out quality–cost trade-offs: 79.60%/0.92vs.73.86%/1.24 on ClawMark, and 62.89%/0.45vs.57.33%/0.61 on GAIA2.

Together, these results provide consistent evidence that ParetoSaddler's improvements are not simply attributable to additional promotions or greater optimization expenditure.

Q2/W4: How much of the gain remains if patches can’t touch app or tool code?

We thank the reviewer for raising this important concern. Following the reviewer's suggestion, we conducted an additional GAIA2 experiment in which only prompts and agent control logic are editable, while simulated applications, tool definitions and implementations remain fixed. We evaluated the resulting harness on the same 300 held-out tasks with three repetitions.

MethodMetricOverall (300)U21 (107)U22 (112)U27 (81)
Default AgentQuality ↑56.22 ± 1.0758.26 ± 1.0853.57 ± 1.5557.20 ± 2.57
Cost ↓0.78 ± 0.080.82 ± 0.180.71 ± 0.030.82 ± 0.04
PS-Pareto$^C$Quality ↑62.11 ± 2.4663.24 ± 4.8059.23 ± 1.0364.61 ± 3.77
Cost ↓0.47 ± 0.000.44 ± 0.010.49 ± 0.020.47 ± 0.00
PS-Pareto$^Q$Quality ↑62.89 ± 2.1461.99 ± 2.3560.71 ± 2.3667.08 ± 1.89
Cost ↓0.45 ± 0.020.39 ± 0.010.49 ± 0.030.46 ± 0.02
PS-Pareto (Fixed Tools)Quality ↑60.78 ± 0.5163.86 ± 1.4353.87 ± 1.3666.26 ± 1.89
Cost ↓0.68 ± 0.020.65 ± 0.010.66 ± 0.030.75 ± 0.02

The results demonstrate that ParetoSaddler improves both quality and cost even without tool modifications. With fixed tools, ParetoSaddler achieves 60.78% quality at $0.68 per task, compared with 56.22% at $0.78 for the default agent, representing a 4.56-percentage-point quality improvement and a 13% reduction in LLM cost. These gains come exclusively from agent-side instructions, such as reusing available information, resolving ambiguities, and improving task execution logic.

The stronger results under full-harness optimization further highlight the value of optimizing tools as part of the harness. As defined in Section 3, an agent harness encompasses prompts, tools, middleware, and control logic. Thus, improving tool interfaces through batched operations, filtered retrieval, and compact output representations is an intentional and legitimate optimization opportunity, particularly for reducing execution costs in long-horizon agentic tasks.

Overall, this additional ablation confirms that ParetoSaddler remains effective with fixed tools, while full-harness optimization unlocks further quality–cost improvements. We will include the above experiment results and discussion into the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer CjhS (2/3)

Q3/W2: How stable are the ablation conclusions across repeated runs?

We thank the reviewer for highlighting the potential impact of optimization-run variability. To assess the robustness of our ablation findings, we conducted an additional independent end-to-end optimization run for each of the three ablated configurations on ClawMark.

ComparisonReferenceAblated Run 1Ablated Run 2
RQ1: Promotion criterionPareto Gate: 75.16 / 0.82MinCost Gate: 73.95 / 0.90MinCost Gate: 68.95 / 0.93
RQ2: Execution repetitions (C)Three-run: 79.60 / 0.92Single-run: 76.95 / 0.95Single-run: 65.27 / 0.74
RQ2: Execution repetitions (Q)Three-run: 81.35 / 1.01Single-run: 75.71 / 0.97Single-run: 66.18 / 0.81
RQ3: Screening rule (C)Bayesian: 79.60 / 0.92Mean: 76.86 / 1.09Mean: 71.70 / 1.06
RQ3: Screening rule (Q)Bayesian: 81.35 / 1.01Mean: 76.53 / 1.15Mean: 76.97 / 1.14

Values are held-out Quality (%) / cost ($ per task). Ablated Run 2 denotes an additional independent optimization run under the same ablation configuration. C/Q denote cost-/quality-oriented endpoints selected on the development set.

The main ablation findings remain consistent across the additional runs:

  • RQ1 (Pareto vs. MinCost gate): Both MinCost runs yield lower quality and higher cost than the Pareto gate, reinforcing the benefit of Pareto-aware promotion.
  • RQ2 (Repeated vs. single execution): Both single-run searches yield lower quality than the three-run setting. Moreover, their quality differs by 11.68 and 9.53 percentage points at the C/Q endpoints, respectively, highlighting the instability of single-run optimization. While the second run achieves lower cost, the results support the value of repeated execution for more reliable optimization.
  • RQ3 (Bayesian vs. mean screening): Both mean-based runs yield lower quality and higher cost than Bayesian screening at both endpoints, reinforcing the benefit of uncertainty-aware promotion.

Overall, all three ablation conclusions remain consistent across the additional independent runs, providing further evidence for the effectiveness of Pareto-aware promotion, repeated execution, and Bayesian screening. We will include these results in the revised manuscript.

W3: Bayesian Screening vs. Simple Robustness Rules

We thank the reviewer for this suggestion. We adopt the Bayesian bootstrap as a lightweight, statistically grounded approach to account for uncertainty in quality–cost observations. To assess its effectiveness against simpler heuristics, we retrospectively evaluated two alternative screening rules using the same candidates, parents, and observations.

  • Worst-task regression guard. Following the reviewer's suggestion, we add a worst-task guard to the mean-based Pareto gate: no individual task's average quality may decrease by more than 10 percentage points relative to its parent. This reproduces all Bayesian decisions on ClawMark but differs on 5/25 candidates on GAIA2. Notably, GAIA2 uses binary scores with three executions per task, making the smallest observable task-level decline 33.3 points. Thus, the 10-point guard effectively vetoes any regression. For example, it rejects I-8 and I-10 because of a single-task regression, although both improve mean screening quality by 11.1 points and establish new lowest-cost development operating points.

  • Leave-one-task-out rule. We also require the mean-based criterion to hold on the full batch and after excluding each screening task in turn. This promotes only 9/17 and 13/25 candidates on ClawMark and GAIA2, respectively, illustrating the greater conservatism of hard robustness constraints.

Overall, these results highlight a key trade-off: hard robustness rules can reject valuable quality–cost improvements due to isolated task regressions, whereas Bayesian screening provides graded statistical support without imposing rigid task-level vetoes.

AuthorsOpenReview ↗

Response to Reviewer CjhS (3/3)

W5: Optimization Expenditure and Amortization

We thank the reviewer for highlighting the importance of optimization expenditure. We now provide a detailed accounting of optimization expenditure across both benchmarks, including optimizer, training, and development evaluation costs, along with estimates of deployment amortization.

ClawMark optimization costs ($)

MethodOptimizer costTrain costDev costTotal optimization cost
AS-Quality168.61484.52960.191,613.32
MH-MinCost41.912608.78-2,650.69
MH-Pareto31.382725.81-2,757.19
PS-MinCost191.26334.95770.791,297.00
PS-Pareto213.07332.95808.301,354.32

GAIA2 optimization costs ($)

MethodOptimizer costTrain costDev costTotal optimization cost
AS-Quality150.24313.002,649.953,113.19
MH-MinCost53.734279.24-4,332.97
MH-Pareto81.254945.55-5,026.80
PS-MinCost200.62211.871390.891803.38
PS-Pareto240.51214.391,903.762,358.66

Key finding: ParetoSaddler achieves stronger held-out quality–cost performance at lower total optimization expenditure. Despite higher optimizer-side costs, PS-Pareto reduces total expenditure relative to AS-Quality by 16.1% on ClawMark (\1,354vs.1,354 vs. \\1,613) and 24.2% on GAIA2 (\2,359vs.2,359 vs. \\3,113). Although ParetoSaddler incurs higher optimizer costs, the increasingly cost-efficient harnesses discovered during optimization reduce subsequent training and development evaluation expenses, particularly in later stages. This demonstrates that optimizing harness execution cost can also improve the efficiency of the optimization process itself.

We further estimate the number of deployed tasks required to recover the optimization expenditure through execution-cost savings relative to the initial harness.

BenchmarkMethodOptimization cost ($)Deployment cost/task ($)Number of Break-even tasks vs. Initial
ClawMarkPS-Pareto$^Q$1354.321.018465
ClawMarkPS-Pareto$^C$1354.320.925418
GAIA2PS-Pareto$^Q$2,358.660.457148
GAIA2PS-Pareto$^C$2,358.660.477609

Across both benchmarks, the optimization expenditure can be amortized after approximately 5.4k–8.5k deployed tasks. For illustration, a shared optimized harness serving 1,000 active users completing five tasks daily would reach break-even in approximately 1–2 days. While actual amortization depends on deployment scale and workload, these estimates suggest that the upfront optimization expenditure can be recovered rapidly in sufficiently high-volume deployments.

We will incorporate these cost breakdowns, amortization estimates, and a visualization of held-out quality–cost performance versus total optimization expenditure into the revised manuscript.


Thank you again for your valuable feedback. We welcome any further questions and discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.