- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Building on AutoSaddler, this paper optimizes an agent harness (the prompts, tools, and control logic around a fixed LLM) for both task quality and token cost. In each iteration, an optimizer LLM patches a parent harness, and both versions are run three times on a small batch of training tasks. The patch is promoted if at least 70% of Bayesian bootstrap draws show higher quality, or lower cost without a large quality drop. A promoted version is evaluated on the full development set and can serve as a parent later. The final preference, such as a cost limit, is applied only at the end, because a patch that looks worse under it may still lead to better versions. On ClawMark and GAIA2, the method reaches higher test quality and lower cost than the initial harness, and does better than the baselines. Ablations examine the promotion rule, repeated runs, and Bayesian screening.
Critical strengths
Strength 1: The method gives clear gains on two benchmarks, measured on unseen task groups
In Table 1, quality on ClawMark rises from 70.86% to 81.35%, and cost per task falls from USD 1.17 to USD 1.01. On GAIA2, quality rises from 56.22% to 62.89%, and cost falls from USD 0.78 to USD 0.45. The selected harnesses also beat the quality-only AutoSaddler baseline on both quality and cost. Because the data are split by scenario (ClawMark) or Universe (GAIA2), these gains are measured on task groups the optimizer never saw, though still within the same benchmark.
Strength 2: The paper shows a case where a quality-only gate fails, and ablates its key design choices
When the parent already solves every screening task, quality cannot go up, so a quality-only gate rejects every patch. In two GAIA2 examples in Appendix C (I-9 and I-17), screening quality stays at 100% while cost drops by about 40%, and both versions also score higher on the development set. So when quality is saturated, cost can still pick out useful patches. The paper also ablates the design: Figure 3 swaps the promotion rule, and Table 3 replaces Bayesian screening with a mean-based gate.
Critical weaknesses
Weakness 1: The gains from Pareto promotion are not separated from the extra evaluation budget
The paper's central claim is that the final preference should not be used to filter candidates during optimization. Two comparisons test this directly, but both also change the evaluation budget.
The first is Figure 3. The Pareto gate promotes 11 candidates and the MinCost gate only 3. Each promotion adds 60 task runs on the development set, while one round of local screening costs 18. Following the accounting in Table 6, the two runs use about 1,020 and 540 task runs, nearly a factor of two, and the Pareto run also has more parents to build on. Yet the test gap is modest: 75.16% vs 73.95% quality, and USD 0.82 vs USD 0.90 cost.
The second is the headline comparison: ParetoSaddler's highest-quality harness beats the quality-only AutoSaddler baseline by 7.49 points on ClawMark and 5.56 on GAIA2. Besides the gate, the two also differ in whether patch generation uses cost information (Appendix A). The number of promotions likely differs too. The Pareto gate takes the union of "quality goes up" and "cost goes down", with no limit on cost increase (ε_C = +∞), so even a patch with no real effect usually passes: its support is about 1 − ½ × ½ = 0.75 when quality and cost are unrelated, and close to 1 when they rise together. At the same γ = 0.7, a quality-only gate gives such a patch about 0.5 and rejects it, and it rejects every patch when all screening tasks are already solved. In the full ClawMark log, 12 of 17 candidates were promoted (Appendix I). But Table 6 only gives an upper bound that assumes every candidate is promoted. The actual number of runs is not reported, and neither is the number of promotions for the quality-only baseline.
So the results show that the full method works, but not yet that the gain comes from Pareto promotion itself rather than from promoting more candidates and spending more evaluation. The I-11 example has the same gap: it shows that a more expensive patch came before the gains at I-14 and I-16, not that those gains needed it. A control with a similar number of promotions or the same budget would answer this, for example a quality-only or cost-only gate with a lower γ that promotes about as many candidates.
Weakness 2: The paper does not say how many tasks it takes to pay back the cost of optimization
Lower cost is one of the paper's main goals, but the cost in Table 1 only counts per-task execution of the final harness, not the cost of finding it. This cost is not small. On GAIA2, 20 of 25 candidates were promoted (Appendix J), which by the accounting in Table 6 means about 4,500 task runs. At USD 0.45 to USD 0.78 per run, this is roughly USD 2,000 to USD 3,500, plus about USD 240 in optimizer cost (Table 11). The final harness saves USD 0.33 per task over the initial one, so by this rough estimate it takes about 7,000 to 11,000 tasks to recover the cost. Compared with the quality-only AutoSaddler, ParetoSaddler also spends more on the optimizer per patch (USD 9.62 vs USD 6.01) and likely promotes more candidates (Weakness 1). This does not mean the method is not worth it. The quality gain has value on its own, and with enough tasks the cost is recovered. But for a paper whose main goals include cost, the full optimization cost and the break-even task count should be reported, together with the deployment scale the method targets.
Questions for authors
-
(Related to Weakness 1) How many candidates were promoted, and how many task runs were actually used, in the quality-only AutoSaddler baseline, in the two runs of Figure 3, and in the main ParetoSaddler runs on both benchmarks?
-
(Related to Weakness 1) Is there a control with a similar number of promotions or the same budget? For example, if the quality-only gate used a lower γ and promoted as many candidates, would it catch up with ParetoSaddler?
-
(Related to Weakness 2) Could you report the full optimization cost of each method on both benchmarks, including optimizer cost and the cost of task runs for screening and development evaluation? Relative to the initial harness and the quality-only AutoSaddler, how many later tasks are needed to recover it?
4 comments
Response to Reviewer b7ZV (2/2)
Q3/W2: Could you report the full optimization cost of each method on both benchmarks, including optimizer cost and the cost of task runs for screening and development evaluation? Relative to the initial harness and the quality-only AutoSaddler, how many later tasks are needed to recover it?
We thank the reviewer for highlighting the importance of optimization expenditure. We now provide a detailed accounting of optimization expenditure across both benchmarks, including optimizer, training, and development evaluation costs, along with estimates of deployment amortization.
ClawMark optimization costs ($)
| Method | Optimizer cost | Train cost | Dev cost | Total optimization cost |
|---|---|---|---|---|
| AS-Quality | 168.61 | 484.52 | 960.19 | 1,613.32 |
| MH-MinCost | 41.91 | 2608.78 | - | 2,650.69 |
| MH-Pareto | 31.38 | 2725.81 | - | 2,757.19 |
| PS-MinCost | 191.26 | 334.95 | 770.79 | 1,297.00 |
| PS-Pareto | 213.07 | 332.95 | 808.30 | 1,354.32 |
GAIA2 optimization costs ($)
| Method | Optimizer cost | Train cost | Dev cost | Total optimization cost |
|---|---|---|---|---|
| AS-Quality | 150.24 | 313.00 | 2,649.95 | 3,113.19 |
| MH-MinCost | 53.73 | 4279.24 | - | 4,332.97 |
| MH-Pareto | 81.25 | 4945.55 | - | 5,026.80 |
| PS-MinCost | 200.62 | 211.87 | 1390.89 | 1803.38 |
| PS-Pareto | 240.51 | 214.39 | 1,903.76 | 2,358.66 |
Key finding: ParetoSaddler achieves stronger held-out quality–cost performance at lower total optimization expenditure. Despite higher optimizer-side costs, PS-Pareto reduces total expenditure relative to AS-Quality by 16.1% on ClawMark (\1,613) and 24.2% on GAIA2 (\3,113). Although ParetoSaddler incurs higher optimizer costs, the increasingly cost-efficient harnesses discovered during optimization reduce subsequent training and development evaluation expenses, particularly in later stages. This demonstrates that optimizing harness execution cost can also improve the efficiency of the optimization process itself.
We further estimate the number of deployed tasks required to recover the optimization expenditure through execution-cost savings relative to the initial harness.
| Benchmark | Method | Optimization cost ($) | Deployment cost/task ($) | Number of Break-even tasks vs. Initial |
|---|---|---|---|---|
| ClawMark | PS-Pareto$^Q$ | 1354.32 | 1.01 | 8465 |
| ClawMark | PS-Pareto$^C$ | 1354.32 | 0.92 | 5418 |
| GAIA2 | PS-Pareto$^Q$ | 2,358.66 | 0.45 | 7148 |
| GAIA2 | PS-Pareto$^C$ | 2,358.66 | 0.47 | 7609 |
Across both benchmarks, the optimization expenditure can be amortized after approximately 5.4k–8.5k deployed tasks. For illustration, a shared optimized harness serving 1,000 active users completing five tasks daily would reach break-even in approximately 1–2 days. While actual amortization depends on deployment scale and workload, these estimates suggest that the upfront optimization expenditure can be recovered rapidly in sufficiently high-volume deployments.
We will incorporate these cost breakdowns, amortization estimates, and a visualization of held-out quality–cost performance versus total optimization expenditure into the revised manuscript.
Thank you again for your valuable feedback. We welcome any further questions and discussion.
Thanks for your response. It addresses all my concerns. I think improving the efficiency to use token is as important as improving the ability to solve tasks. I trust this paper is worth to be accepted. I have raised my score from 3 to 4.
Dear Reviewer b7ZV,
Thank you very much for your encouraging feedback and for reconsidering your assessment. We are delighted that the additional experiments and analyses have addressed your concerns. We sincerely appreciate your constructive suggestions, which have helped us strengthen the paper.
Best regards, Authors
Response to Reviewer b7ZV (1/2)
We thank the reviewer for the encouraging assessment and detailed feedback. We have conducted additional controlled experiments and provided comprehensive optimization-cost accounting to address the concerns about promotion frequency and evaluation expenditure.
We thank the reviewer for this question. The table below summarizes candidate promotions and task-execution counts across all six runs. We will include these details in the revised manuscript.
In the main comparisons, PS-Pareto promotes only two more candidates than AS-Quality on each benchmark (12 vs. 10 on ClawMark; 20 vs. 18 on GAIA2), yet achieves substantially better held-out quality–cost trade-offs. In the Figure 3 ablation, the Pareto gate promotes 11 candidates versus 3 for MinCost, resulting in 1,020 versus 540 total task runs. To disentangle the effects of promotion strategy and evaluation budget, we further conducted promotion-matched and budget-matched controlled experiments, presented in our response to Q2/W1 below.
We thank the reviewer for this valuable suggestion. To isolate the benefits of Pareto-aware promotion from increased promotion frequency or optimization budget, we conducted two additional controlled experiments and analyzed promotion-matched results from our existing runs.
Controlled experiments. We choose the ClawMark MinCost setting (Figure 3) as an example, where the Pareto and MinCost gates differ substantially in promotion frequency (11 vs. 3). We introduce two controls: (1) a random promotion gate that matches the exact number of promotions (also suggested by Reviewer CjhS), and (2) an extended MinCost run that matches the total optimization budget (we found it challenging to guarantee matched promotions via lowering the threshold ). Specifically:
Promotion-matched random gate: Randomly promotes exactly 11 candidates across 17 proposals, matching the number of promoted candidates in Pareto gate.
Budget-matched MinCost gate: Extends MinCost optimization from 17 to 25 iterations, closely matching the Pareto gate's total optimization expenditure (\1,296.99vs.1,290.85).
The held-out ClawMark results are:
The Pareto gate outperforms both controls on quality and cost: by 4.78 percentage points and \0.30/taskoverrandompromotion,andby4.47pointsand0.17/task over budget-matched MinCost. Notably, extending MinCost optimization does not guarantee better generalization: its additional search selects a candidate with improved development performance but worse held-out results. These findings suggest that neither additional promotions nor greater optimization expenditure alone explains the Pareto gate's advantage.
Complementary evidence from PS-Pareto vs. AS-Quality. Across both benchmarks, PS-Pareto promotes only two more candidates than AS-Quality (12 vs. 10 on ClawMark; 20 vs. 18 on GAIA2). Even at matched promotion counts (10 and 18, respectively), PS-Pareto achieves better held-out quality–cost trade-offs: 79.60%/0.92vs.73.86%/1.24 on ClawMark, and 62.89%/0.45vs.57.33%/0.61 on GAIA2.
Together, these results provide consistent evidence that ParetoSaddler's improvements are not simply attributable to additional promotions or greater optimization expenditure. We will incorporate these additional results into the revised manuscript.