- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces ABCAgent to address inconsistent behavior across similar tasks, repeated model-call costs, and difficulties following long workflows. ABCAgent specifies runtime behavior using a symbolic program, such as Python code with neural functions, while a foundation-model agent edits the program for flexibility. The authors evaluate it on six agent benchmarks, including two constructed to test generalization to task variants. They report comparable performance to a model-matched neural agent on GAIA and augmented GAIA, higher accuracy on GSM-Symbolic and τ²-telecom, and lower cost and latency through program reuse on parametric task families.
Critical strengths
- The improvement on τ²-telecom is substantial: Pass⁴ rises from 47.4% to 71.9%. Since this metric requires success in all four trials, the result provides useful evidence about reliability under repeated execution.
- The GSM-Symbolic results show a practical benefit of program reuse: 92.6% of instances are solved without entering the authoring loop, with approximately 7× cost efficiency. This is relevant to recurring tasks that share a stable solution structure.
Critical weaknesses
- The experiments do not rule out the effects of implementation differences. In Table 4, the neural agent baselines and ABCAgent use different harnesses. This comparison therefore cannot establish whether storing behavior as code actually improves the use of historical transcripts. The augmented-GAIA baseline without reuse also runs on an earlier code version, and the GSM-Symbolic transfer comparison uses results from two implementation versions. The authors should provide comparisons using the same harness and code revision.
- The efficiency comparison is not sufficiently rigorous. On augmented GAIA, Table 1 reports a 1.33× speedup, whereas the end-to-end speedup is 0.64×. The cost comparison only covers child requests after the programs have been built, leaving out the additional $257.90 and 70 hours spent on the parent tasks. For augmented GAIA, the authors should report cumulative cost and time, including the parent stage, with both methods processing the same complete request sequence. The corresponding break-even point should also be reported. Results on child requests alone do not establish whether the workload is cheaper overall. For the fallback that invokes the FM Designer to revise programs, the authors should report how reliably failures are identified, how often revisions succeed, and the additional overhead.
- The transfer results do not sufficiently support generalization to new requirements. GSM-Symbolic is well suited to parameter reuse, so these results only demonstrate transfer within a fixed derivation structure. The evidence on augmented GAIA is considerably weaker: forward transfer increases from 30.2% to 33.2%, with p = 0.410. This does not establish improved forward generalization on more complex tasks. Keeping previous tasks as regression cases also does not guarantee that a program can handle unseen requirements. Beyond the reported fraction of tasks solved without rewriting programs, the authors should provide a breakdown and additional experiments distinguishing tasks that require only parameter substitution, tasks that require changes to program logic, and tasks that require a new program.
Questions for authors
- Do the Table 4 and transfer improvements hold with the harness and code revision fixed?
- What are the cumulative cost, end-to-end time, and break-even point on augmented GAIA when both systems process the same parent and child requests?
- How often does rebinding fail, and what are the success rates and costs of revision versus generating a new program?
We thank the reviewer for the constructive and detailed review, and answer the three questions in order.
Q1 (fixed harness and code revision). Table 4. As Appendix B notes, its first two rows ran on a different harness. We added a neural agent that runs in our harness at one code revision and is given the same practice as ABCAgent's authoring: 8 episodes per task (the authoring median), rendered as the author saw them, with the reward withheld. It reaches 0.752 Pass^1 and 0.746 Pass^4, against ABCAgent's 0.816 and 0.719, at about \575 per evaluation pass against ABCAgent's \ 20 (run 1). Averaged over two runs, ABCAgent's Pass^1 advantage is +11.0 points (95% CI [+2.1, +19.7], p = 0.013), and its Pass^4 advantage +2.2 (p = 0.68). The practice information thus accounts for most of the Pass^4 gain, and storing it as a program retains a Pass^1 gain with 30× the cost efficiency per pass. ABCAgent was also stable when the benchmark's simulated user shifted between our two runs: it lost 2 points, the neural agents 10–11. We revised the corresponding sentence in Section 4.
Augmented GAIA without reuse. No conclusion rests on this setting: all three systems tie (58.5%, 57.1% and 59.5%; p = 0.60 and 0.66).
GSM-Symbolic transfer. As Appendix B of the submission states, the two sides were authored at different code revisions, and the reuse side passes a program on only if it solved its own task; both sides' programs are rebound and executed by the same replay script (Table 8). We reran the comparison at one code revision, with no labels anywhere in the reuse chain, on 50 pre-registered families. Forward transfer is 93.4% for ABCAgent against 84.9% for the control of Table 3 (+8.4 points, p = 0.004, significant at our pre-registered α = 0.01). With the variant check also on in the control, the control reaches 90.3%, so reuse itself adds 3.1 points (p = 0.04) and the variant check the other 5.3. Backward, reuse reaches 91.3% against 83.6% and 88.8% (p = 0.27 and 0.91): without labels, the earlier tasks a program is kept consistent with are no longer known to be solved correctly, so Table 3's backward advantage depends on them. We added this run to Appendix B and qualified Section 4 accordingly.
Q2 (cumulative cost, time and break-even on augmented GAIA). Appendix B of the submission reports the parent stage separately. Counting parents and children together, on the API meter and by wall clock, ABCAgent is 0.51× as cost-efficient as the neural agent (about $3,100 against $1,580) and 0.58× as fast (273 h against 158 h). The neural arm has no separate parent run, so its base-GAIA run stands in for it, and for ABCAgent's wall clock as well. Augmented GAIA therefore lies below the break-even point, as Appendix A anticipates for low reuse. The revised Table 1 separates the two components. A rebound child costs about \0.004 and 5 s, against the neural agent's \2.74 and 18 minutes. Authoring a program is 0.37× as cost-efficient as one neural task, and by wall clock 0.51× as fast. A program thus recovers its authoring after about 1.7 reuses in cost and 1 in wall-clock time, whereas augmented GAIA, with three children per parent and 20.1% of them rebound, offers about 0.25 reuses per program. In contrast, GSM-Symbolic (12.5 reuses per program) and τ²-bench (every evaluation pass) are well past it.
Q3 (rebinding failures; revision versus a new program).
Success rate by path, on the same children (ABCAgent / neural):
A rebound child costs about \0.004. An authored child costs \7.31 on average; the meter does not separate revision from rewriting.
Transfer to new requirements. We agree that GSM-Symbolic tests transfer within a fixed derivation. The requested breakdown on augmented GAIA (details in our reply to WMGR):