acceptodds
Under review as a conference paper at ICLR 2027

Divide, Diagnose, and Compose: Optimizing Agentic Pipelines and Harnesses

Abstract

Building agentic pipelines that achieve high task performance while keeping cost and latency low requires jointly optimizing heterogeneous and coupled design choices, including prompts, topology, model selection, and tool policies. Manually searching this combinatorial design space is expensive and time-consuming, while existing automated optimizers typically operate within a predefined subset of this space, modify the pipeline monolithically, or provide little structured evidence for pipeline changes. To mitigate these challenges, we introduce the Decomposed Credit-Assignment Framework (DeCAF), which adapts the design space to the given benchmark and pipeline, optimizes each dimension separately, and generates reasoning traces for every decision. Given a task specification, benchmark examples, and a seed pipeline, DeCAF identifies relevant optimization dimensions and generates diverse experimental hypotheses for each. Independent coding-agent sessions then optimize each dimension under a restricted edit scope, jointly evaluating performance, inference cost, and latency while recording all hypotheses, edits, and outcomes of each experiment. A final synthesis session combines compatible findings across all dimensions and uses a component-level trajectory evaluator, which derives role-specific rubrics from each agent's instructions, to localize underperforming components and guide cross-dimension revisions. Across a diverse suite of agentic benchmarks, DeCAF improves average task performance by 2.58 percentage points, while reducing inference cost by 4.30% and end-to-end latency by 42.45% compared to the best overall baseline.

Primary area:
optimization
Keywords:
Harness Optimization, Agentic Pipeline Builder

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4M6DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies automated optimization of agentic pipelines and harnesses, where prompts, model choices, workflow structure, and context handling interact. It proposes DECAF, which decomposes the design space into task-specific dimensions and explores diverse modifications in separate coding-agent sessions. A final synthesis stage uses component-level trajectory diagnostics to combine and repair these modifications, preserving different performance, cost, and latency trade-offs. Experiments on AppWorld, LiveCodeBench, AgentDojo, and PostTrainBench report higher mean task scores and lower latency than Meta-Harness, with mixed changes in deployment cost; ablations and case studies examine the contributions of the framework’s stages.

Critical strengths
  • The paper addresses a practical problem by optimizing interacting system-level choices rather than restricting optimization to prompts alone.
  • The paper is generally well organized, and the three-stage workflow is clearly illustrated and explained.
  • Reporting deployment performance, cost, and latency alongside optimization overhead supports a practical assessment of the method.
Critical weaknesses
  • Close pipeline optimizers such as MASS and AFlow are discussed but not experimentally compared. Their exclusion needs clearer justification given their overlap with the proposed approach.
  • Only two optimization runs are reported. Although standard errors are provided, the smaller performance differences remain difficult to assess reliably.
  • All optimizers use the same LLM backend, leaving robustness to weaker optimizers or different model families untested.
  • Aggregate improvements coexist with subgroup regressions, including LiveCodeBench Hard and the Gemma PostTrainBench average. These trade-offs deserve clearer discussion.
Questions for authors
  • Why were MASS and AFlow excluded from the experimental comparison? Could they be evaluated within a shared editing space and budget?
  • How sensitive are the GEPA/MASPO comparisons to the scalarization weights and final candidate-selection rule?
  • Could additional optimization runs clarify whether the smaller performance differences are stable?
  • What explains the LiveCodeBench Hard regression and the below-seed Gemma average on PostTrainBench?
  • Does DECAF retain its advantage when using a weaker optimizer model or a different model family?
  • Beyond PostTrainBench, can an optimized harness transfer to new task distributions without additional search?
  • When are diagnosis and synthesis worth their overhead, given their small score contributions on AppWorld and larger contributions on LiveCodeBench?
3 comments

We thank the reviewer for their constructive suggestions. Below, we address each point and specify the changes we made to the manuscript.

Weaknesses

W1: AFlow and MASS search over prompts and workflow topology using predefined building blocks, designed for single-turn tasks. Three of our four benchmarks (AppWorld, AgentDojo, PostTrainBench) are interactive, stateful harnesses, where applying them would require redesigning those building blocks. We therefore chose one recent method per family: GEPA and MASPO for prompt optimization, and Meta-Harness for whole-pipeline search.

AFlow. It applies directly to LiveCodeBench, so we ran it there with the same seed pipeline, data split, and objective as the other baselines. AFlow reaches 50.49% (cost \0.0065,latency38.7s),comparedwith47.330.0065, latency 38.7 s), compared with 47.33% for the seed and 53.16% for DeCAF (\\0.0068, 27.9 s). DeCAF has similar performance with 28% lower latency at similar cost.

MASS. MASS has no public implementation, so we could not run it. We build on their insight that several dimensions should be optimized together (§1).


W2: We agree that the smaller differences are within noise. We could not complete more runs per method on every benchmark because optimization is costly. However, on AgentDojo, where DeCAF’s performance is closest to Meta-Harness, we ran DeCAF two more times. The new runs score 88.09% and 87.44%. The average of all four runs is still 87.77%, and the standard error drops from 0.45 to 0.23. We would like to re-emphasize that although the performance gap to Meta-Harness on AgentDojo is small (1.08 pp), DeCAF has 31% lower latency and 5% lower cost.


W3: To test a weaker optimizer from a different model family, we ran DeCAF and Meta-Harness once each on AgentDojo with MAI-Code-1.1-Flash (medium reasoning effort) in the GitHub Copilot coding agent, in place of Claude-Sonnet-5 in Claude Code. Meta-Harness scores 84.42% and DeCAF 84.85% (one more episode solved). Both remain about 18 pp above the seed (66.71%) and about 2-3 pp below their Sonnet-5 results (86.69% and 87.77%). The two scores are within noise of each other, as with Sonnet-5. DeCAF keeps lower latency (28.96 s vs. 32.08 s) and lower cost (\0.0131vs.0.0131 vs. \\0.0139). While this doesn’t prove generalizability, DeCAF is not tied to one model family or coding agent.


W4: We agree that the gains are not uniform, and we have added this discussion to §5.1 (Line 364).

LiveCodeBench. DeCAF’s gain comes from Medium problems (+15.4 pp over the seed). On Hard it is flat: 38.80% vs. 39.20% for the seed, within the standard error. The change DeCAF kept is a complexity-analysis stage (App. A.2). It helps pick the right algorithm, which is the bottleneck on Medium. Meta-Harness is higher on Hard (42.80%) and lower on Medium (65.43%), so the two methods trade off across difficulty.

PostTrainBench, Gemma. DeCAF’s average is 29.07% vs. 29.98% for the seed, within the standard errors (±0.83 and ±2.65). In exchange, latency drops by 60% (3:10 vs. 7:49) and cost by 12%. DeCAF’s pipeline trains for less time and reserves time for publishing the checkpoint (App. A.4). This costs some HumanEval score on Gemma (30.47% vs. 36.59%), while BFCL improves (56.75% vs. 53.36%).

Questions

Q1: Please see W1.


Q2: GEPA and MASPO optimize Eq. 4 with weights (0.8, 0.1, 0.1), so they mostly optimize task score. On AppWorld and PostTrainBench, DeCAF’s pipeline is better than GEPA’s and MASPO’s on all three metrics, so it is ahead under any weights. The same holds against MASPO on AgentDojo. Against GEPA on AgentDojo, DeCAF is more accurate and cheaper but 6% slower, so it is ahead unless latency is weighted heavily. LiveCodeBench is the sensitive case: DeCAF is more accurate (+4.9 pp), while MASPO’s pipeline is cheaper and slightly faster. There DeCAF is ahead only when the score weight is about 0.8 or more. GEPA and MASPO return the candidate with the highest development-set value of Eq. 4. DeCAF returns the pipeline its LLM picks from the archive.


Q3: Please see W2.


Q4: Please see W4.


Q5: Please see W3.


Q6: Our evidence for transfer is within each benchmark. On PostTrainBench, development and test sets share no base models or benchmarks (§4). We did not test transfer across benchmarks. Each optimized pipeline is specialized to its task. For example, the PostTrainBench pipeline is built for post-training small LLMs on a single H100 within 10 hours, and is not a general training recipe. DeCAF, like the baselines, is meant to be rerun for a new task.


Q7: They are worth it when changes from different dimensions conflict. On AppWorld they did not, so removing synthesis (−0.05 pp) or node_eval (−0.30 pp) changes the score little, though without node_eval latency rises from 32.81 s to 41.61 s. On LiveCodeBench they did conflict, and removing synthesis costs 8.35 pp and removing node_eval 3.06 pp. The overhead is small: the synthesis stage is about 7-11% of DeCAF’s optimization cost, and node_eval about 6% (§5.2, Table 2). Since we cannot tell in advance whether dimensions will conflict on a new benchmark, DeCAF keeps both.

We sincerely appreciate your time and constructive feedback on our paper. As the discussion period is coming to a close, we would kindly request that you review our rebuttal and consider the clarifications provided when reassessing your evaluation. We would be happy to address any remaining concerns.

Thank you again for your time and consideration!

Reviewer_YgzSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DECAF organizes agentic-system optimization into task-adaptive dimensions, independent coding-agent experiments, and a final synthesis stage. A trajectory-based node evaluator guides revisions, while candidates are validated through end-to-end performance, cost, and latency measurements. Experiments on four benchmarks report improved deployment trade-offs over existing optimizers.

Critical strengths
  1. The framework handles interactions pragmatically. It preserves successful and failed experiments, tests composed changes end-to-end, and uses node diagnostics as advisory evidence rather than exact causal credit.
  2. The evaluation is substantial. Diverse agentic benchmarks, OOD post-training evaluation, informative ablations, and downstream re-execution after fault injection provide complementary evidence.
Critical weaknesses
  1. The claimed frontier improvement is not fully established. Results compare one LLM-selected candidate per method rather than complete Pareto fronts or matched operating constraints. With only two optimization runs, the smaller performance differences are also uncertain.

  2. Search-budget comparability needs clarification. DECAF includes dimension searches and additional synthesis, while actual experiment and evaluation counts are unclear. The nine-experiment description is difficult to reconcile with the appendix’s dimension-level experiment menus, and optimization costs differ materially.

Questions for authors
  1. Could you compare Pareto fronts or performance under fixed deployment-cost and latency constraints using a shared selection rule?
  2. Please report actual experiments, full evaluations, and synthesis budgets per method, reconciling the nine-experiment statement with the appendix.
  3. Do the gains persist across more optimization seeds and matched total search budgets?
  4. Does task-adaptive decomposition outperform a fixed, balanced dimension schedule with the same hypotheses and synthesis budget?
3 comments

We thank the reviewer for finding our work valuable and for their constructive feedback. Below, we address each point and specify the changes we made to the manuscript.

Weaknesses

W1: Full Frontier. We agree that the smaller differences are within noise. Originally we ran only each method’s selected pipeline on the test set, because each test pass is costly (e.g, ~\35foronetestrunforAppWorld, 35 for one test run for AppWorld, ~\\60 for PostTrainbench and upto 60 hours for one test run). For this rebuttal, we ran every archive member of DeCAF and Meta-Harness on the LiveCodeBench test set (performance, cost, latency):

Meta-HarnessDeCAF
(47.57, 0.0051, 30.8)(47.57, 0.0051, 30.8)
(45.63, 0.0050, 32.0)(46.12, 0.0049, 31.1)
(48.54, 0.0052, 36.8)(49.51, 0.0054, 27.3)
(51.46, 0.0065, 44.4)(52.91, 0.0068, 27.9)
(58.25, 0.0235, 20.5)(58.25, 0.0216, 20.7)

Against a shared reference point (score 0, and 1.1× the highest cost and latency in either archive, with each axis scaled to 0-1), the hypervolume is 0.204 for DeCAF and 0.160 for Meta-Harness. No DeCAF archive member is dominated by a Meta-Harness member, while one Meta-Harness member is dominated by a DeCAF member. Both archives reach the same top score (58.25%) with a high-cost pipeline, and DeCAF’s version is 8% cheaper. In the low-cost range, DeCAF’s pipelines are much faster (27.3-27.9 s vs. 36.8-44.4 s) at similar score and cost. So DeCAF’s advantage comes from the pipelines its search found, which are mainly faster, and not only from the LLM’s final choice.

Additional Runs. We could not complete more runs per method on every benchmark because optimization is costly. However, on AgentDojo, where DeCAF’s performance is closest to Meta-Harness, we ran DeCAF two more times. The new runs score 88.09% and 87.44%. The average of all four runs is still 87.77%, and the standard error drops from 0.45 to 0.23. We would like to re-emphasize that although the performance gap to Meta-Harness on AgentDojo is small (1.08 pp), DeCAF has 31% lower latency and 5% lower cost.


W2: DeCAF and Meta-Harness have the same planned experiment budget. On AppWorld this is 24 experiments. DeCAF gets 4 dimensions × 6 proposed experiments. For Meta-Harness, N is the number of rounds (4) and k is the number of candidates proposed per round (6).

Not every proposal gets a full evaluation. Both methods can first probe a candidate on a development subset (--dev or --ids), and may then drop or revise it (App. A.1 shows this for DeCAF). Counting probes, DeCAF evaluated 17 candidates and Meta-Harness 20. Of these, 9 per method were run on the full development set. DeCAF’s counts and cost include the synthesis stage. “9 experiments” in §5.1 refers to these 9 full evaluations.

On cost, DeCAF spends about 18% more on average (§5.2). On AppWorld it spent more (\191.86vs.191.86 vs. \\166.91) with fewer candidates, because it tested more candidates that use bigger models, which cost more to evaluate. To check that spend does not explain the gap, we gave Meta-Harness one more round on AppWorld, bringing its cost to \201.06(201.06 (\\9.20 more than DeCAF). It found nothing better and returned the same pipeline. On AgentDojo, DeCAF already spends less than Meta-Harness (\26.48vs.26.48 vs. \\28.74).


Questions

Q1: Please see W1.


Q2: Please see W2. The synthesis stage is about 7-11% of DeCAF’s optimization cost. Meta-Harness has no synthesis stage.


Q3: Please see W1 and W2.


Q4: AppWorld, LiveCodeBench, and AgentDojo cannot show whether adapting helps, because the planner chose the same four dimensions there. We therefore added BFCL (multi-turn) with a single ReAct seed (Qwen3-4B, with 1.7B and 8B as options). Here, the planner proposed three dimensions: LLM backbone, memory across turns, and multi-agent topology. We compared this with the fixed four (prompt, agent configuration, topology, context), each with the same number of hypotheses. Both arms get 5 hypotheses per dimension, so the planner’s split uses fewer in total (15 vs. 20), with the same synthesis stage. Performance of seed: 10.2%, fixed four: 13.8%, DeCAF: 17.6%. The planner’s split gains 7.4 pp over the seed and 3.8 pp over the fixed dimensions.

We sincerely appreciate your time and constructive feedback on our paper. As the discussion period is coming to a close, we would kindly request that you review our rebuttal and consider the clarifications provided when reassessing your evaluation. We would be happy to address any remaining concerns.

Thank you again for your time and consideration!

Thanks for your clarification and experiment. I'll keep my score at 3.

Reviewer_gsfKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DECAF (Decomposed Credit-Assignment Framework), a coding-agent optimizer for LLM agentic pipelines and harnesses that targets task performance, inference cost, and latency together. Divide: an LLM planner proposes benchmark- and pipeline-specific optimization dimensions (in practice prompts, agent configuration, topology, and context handling), and a generator proposes distinct hypotheses per dimension, following Guided Sampling. Diagnose: one Claude Code session per dimension optimizes a copy of the seed pipeline under a soft edit scope, keeps a non-dominated archive, and logs every experiment. Compose: a final session merges compatible findings, helped by node_eval, an LLM judge that scores each component's correctness, compliance with its own prompt, and downstream utilization; the optimizer LLM then picks one pipeline from the archive. On AppWorld, LiveCodeBench, AgentDojo, and PostTrainBench (two runs each), DECAF is reported to beat Meta-Harness by +2.58 pp in average task score, with 4.30% lower inference cost and 42.45% lower latency. Ablations cover two benchmarks, and in a fault-injection study node_eval localizes 77% of injected failures (random baseline: 31.81%).

Critical strengths
  • The search reliably finds faster pipelines, and the paper reports cost and latency next to accuracy. The selected DECAF pipeline has 31–56% lower latency than Meta-Harness's on all four benchmarks (Table 1). On AppWorld it beats every optimizer baseline on all three objectives while keeping the seed's Sonnet-4.6 backbone: 84.23% TGC, \0.1840,and32.8s,againstMeta−Harness′s80.060.1840, and 32.8 s, against Meta-Harness's 80.06%, \0.2237, and 73.8 s.
  • The analysis is transparent. It includes six ablations that report optimizer overhead, appendix accounts of every dimension's tried and rejected experiments, the full prompts, and a controlled fault-injection test of node_eval with counterfactual re-execution and a candid error analysis (Appendix B).
Critical weaknesses
  • The experiments do not separate the benefit of decomposition from search budget, selection policy, and noise. With two runs per method, the gains on LiveCodeBench (+1.46 pp, about 3 of 206 problems) and AgentDojo (+1.08 pp; DECAF 87.77 vs. MASPO 87.76) are within noise. Spend is not matched, experiment parity with Meta-Harness is unclear, and the final pipelines are chosen by an LLM rather than compared as frontiers. Part of the 42.45% latency headline comes from Meta-Harness's selected pipelines being 34–127% slower than the seed on three benchmarks.
  • Two of the three named mechanisms have narrow support. The "task-adaptive" planner produced the same four dimensions on three of four benchmarks and is never compared with a fixed decomposition. Synthesis with node_eval changes task score materially only on LiveCodeBench (−3.06 pp without node_eval). On AppWorld, removing synthesis changes TGC by −0.05 pp and removing node_eval by −0.30 pp, although the latter raises latency from 32.81 s to 41.61 s. node_eval is validated only on synthetic single faults in successful trajectories, against a random baseline, and without the failure-attribution baselines the paper cites.
Questions for authors
  • How many candidate pipelines did each method evaluate on the dev set, including --dev/--ids probes, and what do Meta-Harness's N and k control? Please reconcile "both Meta-Harness and DECAF run 9 experiments" (5.1) with four dimensions × six experiments (App. A.1) and with Figure 3
  • Can you report a spend-matched comparison, either Meta-Harness given DECAF's per-benchmark dollar budget or both stopped at a common budget, with at least five optimization runs per method and paired per-instance tests or bootstrap CIs on the test-set differences?
  • What are the test-set scores of all archive members for DECAF and Meta-Harness, and what is each archive's test-set hypervolume against a shared reference point? This would separate search quality from the LLM's final choice.
  • How does DECAF compare with a fixed generic decomposition (prompt, agent configuration, topology, context; no planner)? What few-shot decomposition examples does the planner receive? Appendix D.1 shows only a placeholder.
  • How is the Table 1 AgentDojo score computed? It matches the unweighted mean of Under attack, Benign, and Attacked in Table 5. Do dev and test share attack types and injection tasks, and do the gains hold under an attack withheld from development?
  • How are cost (including for self-hosted models) and latency measured: pricing, hardware, concurrency, and dates?
4 comments

We thank the reviewer for the careful read and constructive suggestions. Below, we address each point and specify the changes we made to the manuscript.

Weaknesses

W1: We address each concern below:

Decomposition effect. The No Dimensions ablation (§5.3) keeps the sessions, the number of hypotheses, and the synthesis stage, and removes only the dimension split. Accuracy drops 5.06 pp on AppWorld and 3.23 pp on LiveCodeBench. On LiveCodeBench the two also spend the same (\80.59vs.80.59 vs. \\81.88). This provides more direct evidence for decomposition than the Meta-Harness comparison alone.

Score Differences and Noise. We agree that on LiveCodeBench and AgentDojo, the performance gains are not statistically significant. The clearer advantage is latency. E.g., On LiveCodeBench, it has 37.2% lower latency than Meta-Harness at similar cost, and 31% lower latency with 5% lower cost on AgentDojo. We have revised the results to emphasize this distinction (§5.1).

Optimization Budget. We configured Meta-Harness for similar planned experiment budgets. This does not guarantee equal spending, because candidates that use bigger models cost more to evaluate, and optimizers can propose additional experiments during search. DeCAF costs 18% more to optimize on average (§5.2), but this varies by benchmark. On AgentDojo, DeCAF costs less than Meta-Harness (\26.48vs.26.48 vs. \\28.74). Furthermore, on AppWorld, we gave Meta-Harness one more round, bringing its optimization cost to \201.06(201.06 (\\9.2 higher than DeCAF). It found nothing better and returned the same pipeline.

Selected Pipelines and Frontiers. Both DeCAF and Meta-Harness receive the three objectives, retain non-dominated candidates, and use the same LLM step to select a final pipeline from development-set results, so the selection is the same for both. Our main goal was to find a single optimized pipeline. However, we now also report full frontiers for the two methods on LiveCodeBench (see Q3).

Latency. Even compared to the seed pipeline, DeCAF is faster (−40% AppWorld, −5% LiveCodeBench, +56% AgentDojo, −52% PostTrainBench). On AgentDojo it is 56% slower, in exchange for raising performance from 66.71% to 87.77%, and it is still faster than Meta-Harness.


W2: Task-adaptive planner. AppWorld, LiveCodeBench, and AgentDojo cannot show whether adapting helps, because the planner chose the same four dimensions there. We therefore added BFCL (multi-turn) with a single ReAct seed (Qwen3-4B, with 1.7B and 8B as options). Here, the planner proposed three dimensions: LLM backbone, memory across turns, and multi-agent topology. We compared this with the fixed four (prompt, agent configuration, topology, context). Performance of seed: 10.2%, fixed four: 13.8%, DeCAF: 17.6%. The planner’s split gains 7.4 pp over the seed and 3.8 pp above fixed dimensions.

Synthesis and node_eval. The final synthesis stage matters when changes from different dimensions conflict. On AppWorld they did not, so removing synthesis (−0.05 pp) or node_eval (−0.30 pp) changes the score little. On LiveCodeBench they did conflict, and removing synthesis costs 8.35 pp and removing node_eval 3.06 pp. Since we cannot tell in advance whether dimensions will conflict on a new benchmark, DeCAF keeps both.

node_eval validation. node_eval provides diagnostics rather than exact causal attribution, and candidates are accepted based on measured dev-set task performance, cost, and latency (§3.4). Nevertheless, diagnostic errors can affect the search trajectory and final result. Our ablations evaluate its downstream utility within synthesis, while the controlled single-fault study in Appendix B evaluates localization under known synthetic interventions. We have not compared against dedicated failure-attribution methods and therefore do not claim superior localization accuracy.

Questions

Q1: On AppWorld (App. A.1), both methods have the same planned budget of 24 experiments. DeCAF gets 4 dimensions × 6 proposed experiments. For Meta-Harness, N is the number of rounds (4) and k is the number of candidates proposed per round (6).

Not every proposal gets a full evaluation. Both methods can first probe a candidate on a development subset (--dev or --ids), and may then drop or revise it (App. A.1 shows this for DeCAF). Counting probes, DeCAF evaluated 17 candidates and Meta-Harness 20. Of these, 9 per method were run on the full development set. DeCAF tested more candidates that use bigger models, which cost more to evaluate. This is why it spent more (\191.86vs.191.86 vs. \\166.91) with fewer candidates.

“9 experiments” in §5.1 and the points in Fig. 3 refer to these 9 full evaluations. We have fixed the wording in both places.


Q2: As described in W1. Optimization Budget., on AppWorld, we gave Meta-Harness more than DeCAF’s budget and it returned the same pipeline. On AgentDojo, DeCAF already spends less than Meta-Harness.

We could not complete five runs per method on every benchmark because optimization is costly. However, on AgentDojo, where DeCAF’s performance is closest to Meta-Harness, we ran DeCAF two more times. The new runs score 88.09% and 87.44%. The average of all four runs is still 87.77%, and the standard error drops from 0.45 to 0.23. We would like to re-emphasize that although the performance gap to Meta-Harness on AgentDojo is small (1.08 pp), DeCAF has 31% lower latency and 5% lower cost.


Q3: Originally we ran only each method’s selected pipeline on the test set, because each test pass is costly (e.g, ~\35foronetestrunforAppWorld, 35 for one test run for AppWorld, ~\\60 for PostTrainbench and up to 60 hours for one test run). For this rebuttal, we ran every archive member of DeCAF and Meta-Harness on the LiveCodeBench test set (performance, cost, latency):

Meta-HarnessDeCAF
(47.57, 0.0051, 30.8)(47.57, 0.0051, 30.8)
(45.63, 0.0050, 32.0)(46.12, 0.0049, 31.1)
(48.54, 0.0052, 36.8)(49.51, 0.0054, 27.3)
(51.46, 0.0065, 44.4)(52.91, 0.0068, 27.9)
(58.25, 0.0235, 20.5)(58.25, 0.0216, 20.7)

Against a shared reference point (score 0, and 1.1× the highest cost and latency in either archive, with each axis scaled to 0-1), the hypervolume is 0.204 for DeCAF and 0.160 for Meta-Harness. No DeCAF archive member is dominated by a Meta-Harness member, while one Meta-Harness member is dominated by a DeCAF member. Both archives reach the same top score (58.25%) with a high-cost pipeline, and DeCAF’s version is 8% cheaper. In the low-cost range, DeCAF’s pipelines are much faster (27.3-27.9 s vs. 36.8-44.4 s) at similar score and cost. So DeCAF’s advantage comes from the pipelines its search found, which are mainly faster, and not only from the LLM’s final choice. We have updated these results in Appendix A.2.1.


Q4: On AppWorld, LiveCodeBench, and AgentDojo the planner chose exactly these four dimensions, so the fixed split and the planner’s split are the same there. They differ on the new BFCL (multi-turn) benchmark, where the planner chose a different set (LLM backbone, memory across turns, multi-agent topology). Seed performance: 10.2%, fixed four: 13.8%, DeCAF with the planner 17.6% (see W2).

The planner receives 3 worked decomposition examples from MATH, WebShop, and BuildBench. We have updated the prompt in Appendix D.1 to include them as well.


Q5: Yes. The AgentDojo score in Table 1 is the unweighted mean of Under attack, Benign, and Attacked. Dev has 20 user tasks (60 episodes) and test has 77 (231 episodes). They share the attack types but not the user tasks or the injection tasks. So the test score measures transfer to new user tasks and new attacker goals under the same attack type.


Q6: All models, including the open-weight ones, were called through APIs, none were self-hosted. Cost is the token usage reported by the API times the provider’s list price at the time of the runs. For latency, we time each evaluation from start to end and divide by the number of instances. Instances run one at a time, with the same setting for every method.

We sincerely appreciate your time and constructive feedback on our paper. As the discussion period is coming to a close, we would kindly request that you review our rebuttal and consider the clarifications provided when reassessing your evaluation. We would be happy to address any remaining concerns.

Thank you again for your time and consideration!

Reviewer_gsfKOpenReview ↗

Replying to comments by authors

I thank the authors for their detailed explanations. Most of my concerns about gains being within noise are not yet addressed and still keep my reservations on the methodology. However, the authors do make a fair comparison regarding the cost, latency optimizations. This works provides a strong foundations for optimizing agentic harness training in the future and potentially improving the gains as well. I've raised my scores. Thanks!

Comments on the paper

1 comment

We thank the reviewers for their careful and constructive reviews. We posted our responses and a revised manuscript on November 14. As the discussion period closes today and we have not yet received follow-up comments on these responses, we summarize them here so that they can be considered in the final decision.

1. Reliability of the smaller gains (gsfK, YgzS, 4M6D). We agree that the task-score gains over Meta-Harness on LiveCodeBench and AgentDojo are within noise, and we revised §5.1 to make this distinction. The clearer advantage is latency: 37.2% lower than Meta-Harness on LiveCodeBench at similar cost, and 31% lower with 5% lower cost on AgentDojo. Two additional DeCAF runs on AgentDojo (88.09%, 87.44%) leave the four-run mean at 87.77% and reduce the standard error from 0.45 to 0.23.

2. Search budget and experiment counts (gsfK, YgzS). Both methods have the same planned budget (AppWorld: 24 experiments; 4 dimensions × 6 for DeCAF, 4 rounds × 6 candidates for Meta-Harness). Counting subset probes, DeCAF evaluated 17 candidates and Meta-Harness 20, and each ran 9 on the full development set. DeCAF spends about 18% more on optimization on average, but an extra Meta-Harness round on AppWorld, which put its spend above DeCAF's, returned the same pipeline. On AgentDojo, DeCAF already spends less.

3. Frontiers rather than one selected pipeline (gsfK, YgzS). Both methods use the same LLM selection step. We also ran every archive member of both methods on the LiveCodeBench test set (Appendix A.2.1). Against a shared reference point, hypervolume is 0.204 for DeCAF and 0.160 for Meta-Harness. No DeCAF member is dominated by a Meta-Harness member, and one Meta-Harness member is dominated by a DeCAF member. In the low-cost range, DeCAF's pipelines are faster (27.3–27.9 s vs. 36.8–44.4 s) at similar score and cost.

4. Baselines and optimizer robustness (4M6D). AFlow and MASS target single-turn tasks, while three of our four benchmarks are interactive, stateful harnesses. AFlow applies directly to LiveCodeBench, where it reaches 50.49% at 38.7 s, vs. 53.16% at 27.9 s for DeCAF at similar cost; MASS has no public implementation. With a weaker optimizer from a different model family and coding agent (one run each on AgentDojo), DeCAF and Meta-Harness score 84.85% and 84.42%, within noise of each other, and DeCAF keeps lower latency and cost.


We would be grateful if these responses and the revised manuscript could be taken into account in the discussion and final recommendation, and we remain available to answer any questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.