- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies automated optimization of agentic pipelines and harnesses, where prompts, model choices, workflow structure, and context handling interact. It proposes DECAF, which decomposes the design space into task-specific dimensions and explores diverse modifications in separate coding-agent sessions. A final synthesis stage uses component-level trajectory diagnostics to combine and repair these modifications, preserving different performance, cost, and latency trade-offs. Experiments on AppWorld, LiveCodeBench, AgentDojo, and PostTrainBench report higher mean task scores and lower latency than Meta-Harness, with mixed changes in deployment cost; ablations and case studies examine the contributions of the framework’s stages.
Critical strengths
- The paper addresses a practical problem by optimizing interacting system-level choices rather than restricting optimization to prompts alone.
- The paper is generally well organized, and the three-stage workflow is clearly illustrated and explained.
- Reporting deployment performance, cost, and latency alongside optimization overhead supports a practical assessment of the method.
Critical weaknesses
- Close pipeline optimizers such as MASS and AFlow are discussed but not experimentally compared. Their exclusion needs clearer justification given their overlap with the proposed approach.
- Only two optimization runs are reported. Although standard errors are provided, the smaller performance differences remain difficult to assess reliably.
- All optimizers use the same LLM backend, leaving robustness to weaker optimizers or different model families untested.
- Aggregate improvements coexist with subgroup regressions, including LiveCodeBench Hard and the Gemma PostTrainBench average. These trade-offs deserve clearer discussion.
Questions for authors
- Why were MASS and AFlow excluded from the experimental comparison? Could they be evaluated within a shared editing space and budget?
- How sensitive are the GEPA/MASPO comparisons to the scalarization weights and final candidate-selection rule?
- Could additional optimization runs clarify whether the smaller performance differences are stable?
- What explains the LiveCodeBench Hard regression and the below-seed Gemma average on PostTrainBench?
- Does DECAF retain its advantage when using a weaker optimizer model or a different model family?
- Beyond PostTrainBench, can an optimized harness transfer to new task distributions without additional search?
- When are diagnosis and synthesis worth their overhead, given their small score contributions on AppWorld and larger contributions on LiveCodeBench?
3 comments
Questions
Q1: Please see W1.
Q2: GEPA and MASPO optimize Eq. 4 with weights (0.8, 0.1, 0.1), so they mostly optimize task score. On AppWorld and PostTrainBench, DeCAF’s pipeline is better than GEPA’s and MASPO’s on all three metrics, so it is ahead under any weights. The same holds against MASPO on AgentDojo. Against GEPA on AgentDojo, DeCAF is more accurate and cheaper but 6% slower, so it is ahead unless latency is weighted heavily. LiveCodeBench is the sensitive case: DeCAF is more accurate (+4.9 pp), while MASPO’s pipeline is cheaper and slightly faster. There DeCAF is ahead only when the score weight is about 0.8 or more. GEPA and MASPO return the candidate with the highest development-set value of Eq. 4. DeCAF returns the pipeline its LLM picks from the archive.
Q3: Please see W2.
Q4: Please see W4.
Q5: Please see W3.
Q6: Our evidence for transfer is within each benchmark. On PostTrainBench, development and test sets share no base models or benchmarks (§4). We did not test transfer across benchmarks. Each optimized pipeline is specialized to its task. For example, the PostTrainBench pipeline is built for post-training small LLMs on a single H100 within 10 hours, and is not a general training recipe. DeCAF, like the baselines, is meant to be rerun for a new task.
Q7: They are worth it when changes from different dimensions conflict. On AppWorld they did not, so removing synthesis (−0.05 pp) or node_eval (−0.30 pp) changes the score little, though without node_eval latency rises from 32.81 s to 41.61 s. On LiveCodeBench they did conflict, and removing synthesis costs 8.35 pp and removing node_eval 3.06 pp. The overhead is small: the synthesis stage is about 7-11% of DeCAF’s optimization cost, and node_eval about 6% (§5.2, Table 2). Since we cannot tell in advance whether dimensions will conflict on a new benchmark, DeCAF keeps both.
We sincerely appreciate your time and constructive feedback on our paper. As the discussion period is coming to a close, we would kindly request that you review our rebuttal and consider the clarifications provided when reassessing your evaluation. We would be happy to address any remaining concerns.
Thank you again for your time and consideration!
We thank the reviewer for their constructive suggestions. Below, we address each point and specify the changes we made to the manuscript.
Weaknesses
W1: AFlow and MASS search over prompts and workflow topology using predefined building blocks, designed for single-turn tasks. Three of our four benchmarks (AppWorld, AgentDojo, PostTrainBench) are interactive, stateful harnesses, where applying them would require redesigning those building blocks. We therefore chose one recent method per family: GEPA and MASPO for prompt optimization, and Meta-Harness for whole-pipeline search.
AFlow. It applies directly to LiveCodeBench, so we ran it there with the same seed pipeline, data split, and objective as the other baselines. AFlow reaches 50.49% (cost \0.0065,latency38.7s),comparedwith47.330.0068, 27.9 s). DeCAF has similar performance with 28% lower latency at similar cost.
MASS. MASS has no public implementation, so we could not run it. We build on their insight that several dimensions should be optimized together (§1).
W2: We agree that the smaller differences are within noise. We could not complete more runs per method on every benchmark because optimization is costly. However, on AgentDojo, where DeCAF’s performance is closest to Meta-Harness, we ran DeCAF two more times. The new runs score 88.09% and 87.44%. The average of all four runs is still 87.77%, and the standard error drops from 0.45 to 0.23. We would like to re-emphasize that although the performance gap to Meta-Harness on AgentDojo is small (1.08 pp), DeCAF has 31% lower latency and 5% lower cost.
W3: To test a weaker optimizer from a different model family, we ran DeCAF and Meta-Harness once each on AgentDojo with MAI-Code-1.1-Flash (medium reasoning effort) in the GitHub Copilot coding agent, in place of Claude-Sonnet-5 in Claude Code. Meta-Harness scores 84.42% and DeCAF 84.85% (one more episode solved). Both remain about 18 pp above the seed (66.71%) and about 2-3 pp below their Sonnet-5 results (86.69% and 87.77%). The two scores are within noise of each other, as with Sonnet-5. DeCAF keeps lower latency (28.96 s vs. 32.08 s) and lower cost (\0.0131vs.0.0139). While this doesn’t prove generalizability, DeCAF is not tied to one model family or coding agent.
W4: We agree that the gains are not uniform, and we have added this discussion to §5.1 (Line 364).
LiveCodeBench. DeCAF’s gain comes from Medium problems (+15.4 pp over the seed). On Hard it is flat: 38.80% vs. 39.20% for the seed, within the standard error. The change DeCAF kept is a complexity-analysis stage (App. A.2). It helps pick the right algorithm, which is the bottleneck on Medium. Meta-Harness is higher on Hard (42.80%) and lower on Medium (65.43%), so the two methods trade off across difficulty.
PostTrainBench, Gemma. DeCAF’s average is 29.07% vs. 29.98% for the seed, within the standard errors (±0.83 and ±2.65). In exchange, latency drops by 60% (3:10 vs. 7:49) and cost by 12%. DeCAF’s pipeline trains for less time and reserves time for publishing the checkpoint (App. A.4). This costs some HumanEval score on Gemma (30.47% vs. 36.59%), while BFCL improves (56.75% vs. 53.36%).