acceptodds
Under review as a conference paper at ICLR 2027

Agent Behavior as Code: Efficient and Robust LLM Agents with Programmatic Specifications

Abstract

AI agents based on foundation models (FMs) have demonstrated strong capabilities to perform complex open-ended tasks. However, they face some common challenges in practice: (a) agent behavior can deviate drastically even for semantically similar tasks, leading to catastrophically propagated errors; (b) high cost and latency due to FM calls, repeated in full whenever a task recurs with different inputs; (c) FMs' limited context and instruction following capability confine how well agents manage the ever-growing execution context and follow complex plans. We introduce **A**gent **B**ehavior as **C**ode **Agent** (ABCAgent), which uses a symbolic program (\eg, Python code with potential neural functions) to fully specify the agent's behavior at runtime, with a powerful FM agent editing that program for flexibility. Behavior is thus specified without premature variable binding, and its execution is deterministic. We evaluate ABCAgent on six agent benchmarks, two of which we construct to test how well a derived program generalizes to variants of the task it was written for. ABCAgent matches a model-matched neural agent on GAIA and augmented GAIA, and surpasses it where robustness and long control flows matter: 98.3% against 97.3% on GSM-Symbolic (), 71.9% against 47.4% on the telecom domain of -bench (), and more records written correctly at every loop length on our control-flow-augmented WorkArena benchmark. For more parametric task families, ABCAgent is also significantly superior in efficiency. Without authoring a new program, ABCAgent solves 92.6% of GSM-Symbolic instances and 20.1% of augmented GAIA variants, which yields lower latency and lower cost on GSM-Symbolic, 19% lower cost on augmented GAIA, and lower agent latency on -telecom.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Neuro-symbolic AI, Agent Robustness, Efficient AI Agents

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_8x6gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ABCAgent to address inconsistent behavior across similar tasks, repeated model-call costs, and difficulties following long workflows. ABCAgent specifies runtime behavior using a symbolic program, such as Python code with neural functions, while a foundation-model agent edits the program for flexibility. The authors evaluate it on six agent benchmarks, including two constructed to test generalization to task variants. They report comparable performance to a model-matched neural agent on GAIA and augmented GAIA, higher accuracy on GSM-Symbolic and τ²-telecom, and lower cost and latency through program reuse on parametric task families.

Critical strengths
  1. The improvement on τ²-telecom is substantial: Pass⁴ rises from 47.4% to 71.9%. Since this metric requires success in all four trials, the result provides useful evidence about reliability under repeated execution.
  2. The GSM-Symbolic results show a practical benefit of program reuse: 92.6% of instances are solved without entering the authoring loop, with approximately 7× cost efficiency. This is relevant to recurring tasks that share a stable solution structure.
Critical weaknesses
  1. The experiments do not rule out the effects of implementation differences. In Table 4, the neural agent baselines and ABCAgent use different harnesses. This comparison therefore cannot establish whether storing behavior as code actually improves the use of historical transcripts. The augmented-GAIA baseline without reuse also runs on an earlier code version, and the GSM-Symbolic transfer comparison uses results from two implementation versions. The authors should provide comparisons using the same harness and code revision.
  2. The efficiency comparison is not sufficiently rigorous. On augmented GAIA, Table 1 reports a 1.33× speedup, whereas the end-to-end speedup is 0.64×. The cost comparison only covers child requests after the programs have been built, leaving out the additional $257.90 and 70 hours spent on the parent tasks. For augmented GAIA, the authors should report cumulative cost and time, including the parent stage, with both methods processing the same complete request sequence. The corresponding break-even point should also be reported. Results on child requests alone do not establish whether the workload is cheaper overall. For the fallback that invokes the FM Designer to revise programs, the authors should report how reliably failures are identified, how often revisions succeed, and the additional overhead.
  3. The transfer results do not sufficiently support generalization to new requirements. GSM-Symbolic is well suited to parameter reuse, so these results only demonstrate transfer within a fixed derivation structure. The evidence on augmented GAIA is considerably weaker: forward transfer increases from 30.2% to 33.2%, with p = 0.410. This does not establish improved forward generalization on more complex tasks. Keeping previous tasks as regression cases also does not guarantee that a program can handle unseen requirements. Beyond the reported fraction of tasks solved without rewriting programs, the authors should provide a breakdown and additional experiments distinguishing tasks that require only parameter substitution, tasks that require changes to program logic, and tasks that require a new program.
Questions for authors
  1. Do the Table 4 and transfer improvements hold with the harness and code revision fixed?
  2. What are the cumulative cost, end-to-end time, and break-even point on augmented GAIA when both systems process the same parent and child requests?
  3. How often does rebinding fail, and what are the success rates and costs of revision versus generating a new program?
1 comment

We thank the reviewer for the constructive and detailed review, and answer the three questions in order.

Q1 (fixed harness and code revision). Table 4. As Appendix B notes, its first two rows ran on a different harness. We added a neural agent that runs in our harness at one code revision and is given the same practice as ABCAgent's authoring: 8 episodes per task (the authoring median), rendered as the author saw them, with the reward withheld. It reaches 0.752 Pass^1 and 0.746 Pass^4, against ABCAgent's 0.816 and 0.719, at about \575 per evaluation pass against ABCAgent's \ 20 (run 1). Averaged over two runs, ABCAgent's Pass^1 advantage is +11.0 points (95% CI [+2.1, +19.7], p = 0.013), and its Pass^4 advantage +2.2 (p = 0.68). The practice information thus accounts for most of the Pass^4 gain, and storing it as a program retains a Pass^1 gain with 30× the cost efficiency per pass. ABCAgent was also stable when the benchmark's simulated user shifted between our two runs: it lost 2 points, the neural agents 10–11. We revised the corresponding sentence in Section 4.

Augmented GAIA without reuse. No conclusion rests on this setting: all three systems tie (58.5%, 57.1% and 59.5%; p = 0.60 and 0.66).

GSM-Symbolic transfer. As Appendix B of the submission states, the two sides were authored at different code revisions, and the reuse side passes a program on only if it solved its own task; both sides' programs are rebound and executed by the same replay script (Table 8). We reran the comparison at one code revision, with no labels anywhere in the reuse chain, on 50 pre-registered families. Forward transfer is 93.4% for ABCAgent against 84.9% for the control of Table 3 (+8.4 points, p = 0.004, significant at our pre-registered α = 0.01). With the variant check also on in the control, the control reaches 90.3%, so reuse itself adds 3.1 points (p = 0.04) and the variant check the other 5.3. Backward, reuse reaches 91.3% against 83.6% and 88.8% (p = 0.27 and 0.91): without labels, the earlier tasks a program is kept consistent with are no longer known to be solved correctly, so Table 3's backward advantage depends on them. We added this run to Appendix B and qualified Section 4 accordingly.

Q2 (cumulative cost, time and break-even on augmented GAIA). Appendix B of the submission reports the parent stage separately. Counting parents and children together, on the API meter and by wall clock, ABCAgent is 0.51× as cost-efficient as the neural agent (about $3,100 against $1,580) and 0.58× as fast (273 h against 158 h). The neural arm has no separate parent run, so its base-GAIA run stands in for it, and for ABCAgent's wall clock as well. Augmented GAIA therefore lies below the break-even point, as Appendix A anticipates for low reuse. The revised Table 1 separates the two components. A rebound child costs about \0.004 and 5 s, against the neural agent's \2.74 and 18 minutes. Authoring a program is 0.37× as cost-efficient as one neural task, and by wall clock 0.51× as fast. A program thus recovers its authoring after about 1.7 reuses in cost and 1 in wall-clock time, whereas augmented GAIA, with three children per parent and 20.1% of them rebound, offers about 0.25 reuses per program. In contrast, GSM-Symbolic (12.5 reuses per program) and τ²-bench (every evaluation pass) are well past it.

Q3 (rebinding failures; revision versus a new program).

  • GSM-Symbolic. Rebinding was accepted on 4,631 of 4,900 attempts (94.5%), and 99.0% of those answers are correct.
  • Augmented GAIA. Rebinding solved 76 of 378 children (20.1%; Table 1 setting), and its plausibility check accepted 29 wrong answers. Replayed on 303 unseen children and scored against the labels, the reuse path rejects 86% of the children it would answer wrongly and keeps 87% of its correct answers (details in our reply to WMGR).

Success rate by path, on the same children (ABCAgent / neural):

pathnABCAgentneural
rebinding7661.864.5
program revised (≥50% kept)24059.660.0
program rewritten5048.054.0
no program100.030.0

A rebound child costs about \0.004. An authored child costs \7.31 on average; the meter does not separate revision from rewriting.

Transfer to new requirements. We agree that GSM-Symbolic tests transfer within a fixed derivation. The requested breakdown on augmented GAIA (details in our reply to WMGR):

  • On 222 children that need only new parameter values, ABCAgent matches the neural agent.
  • On 142 that need changed logic, it trails (50.0% against 57.7%), mostly because of the open-ended reuse check.
Reviewer_JS1tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses inconsistent LLM agent behavior across similar tasks, high execution costs, and limitations in long-context management. To tackle these challenges, the authors propose ABCAGENT, which represents agent behavior as executable programs and separates agent design from execution. Specifically, an FM Designer Agent generates and revises programs, while a Symbolic Executor executes their control logic and invokes LLMs when necessary. To enable reliable reuse, Parameter Rebinding adapts existing programs through parameter substitution, while Variant Check evaluates their generalization using generated task variants. When reuse fails, the designer revises the program based on execution feedback while preserving compatibility with previously solved tasks. Experiments across multiple agent benchmarks demonstrate improvements in execution robustness, cost, and latency, particularly for recurring tasks and complex control flows.

Critical strengths
  1. This paper addresses the challenges of unstable LLM agent behavior and execution inefficiency in recurring tasks by proposing a framework that represents agent behavior as executable programs. By separating program design from execution and incorporating parameter reuse and program verification mechanisms, the framework effectively reduces redundant LLM calls and improves execution reliability in complex control flows.

  2. The paper evaluates the proposed method across multiple agent benchmarks, demonstrating significant performance and efficiency improvements, particularly on GSM-Symbolic and τ²-telecom. In addition, the authors provide component ablations, program transfer experiments, and cost analyses that help explain the contributions of different mechanisms.

Critical weaknesses
  1. Limited technical novelty. Programmatic agent control has been explored in CodeAct, Programmatic Skills, and LLM-as-code. Parameter Rebinding and Variant Check also build on established ideas. The lack of comparisons with existing programmatic agents makes it difficult to assess the additional contributions.

  2. Insufficient evidence for generalization and efficiency. Table 3 shows no significant Forward Generalization improvement on Augmented GAIA (33.2% vs. 30.2%, p=0.410). Table 1 shows substantially worse performance and higher costs on WorkArena L1, while Figure 3 indicates that efficiency gains heavily depend on program reuse.

Questions for authors
  1. What are the key technical differences between the proposed method and existing approaches such as CodeAct, Programmatic Skills, and especially LLM-as-code? Could the authors provide direct experimental comparisons with related Programmatic Agent methods to demonstrate the additional benefits of Parameter Rebinding and Variant Check?

  2. On Augmented GAIA, Program Reuse does not significantly improve Forward Generalization. What do the authors consider to be the main bottleneck? For tasks that cannot be solved through simple parameter substitution and instead require modifications to the program logic, can ABCAGENT still achieve consistent generalization improvements?

  3. Some of the reported efficiency metrics do not fully account for all overhead associated with program generation, verification, and execution. Could the authors provide a more comprehensive end-to-end efficiency comparison that includes all these costs, and analyze the actual cost benefits of ABCAGENT over the Neural Agent under different program reuse frequencies?

2 comments

We thank the reviewer for the questions, which we answer in order.

Q1 (differences and comparisons). Section 5 discusses the methods named:

  • CodeAct generates transient code snippets that are discarded after each task.
  • Skill-induction methods store functions, but the FM still orchestrates the top-level control flow.
  • LLM-as-Code is concurrent work, and also lets a program govern control flow.

An ABCAgent program instead specifies the whole agent, declares rebindable parameters, and must pass an executable check of its generality before it is stored (Section 2). The additional benefit of rebinding and the variant check is measured by the ablations in Tables 2, 9 and 10:

  • Rebinding is the only component that reduces cost (Table 9), from 0.96× to 11.91× the neural agent's cost efficiency on GSM-Symbolic (Table 2).
  • On WorkArena-CF, programs stored without the variant check write no record correctly when the executor replays them (Table 10).

Our neural baseline, Claude Code, itself acts by writing and executing code. As an in-harness analogue of LATM, we froze ABCAgent's GSM-Symbolic program after at most 4 instances and applied it without the check or the fallback. It solves 95.4% of 4,637 later instances against 99.1% for full ABCAgent (+3.6 points, 95% CI [+1.8, +5.8]). For the final version, we will add LATM and a programmatic skill-induction agent (ASI) in our harness, with the same model and meter, on the benchmarks their published designs support (few-shot reasoning; web navigation).

Q2 (bottleneck on augmented GAIA). We classified each child by what its parent's program would need (a model shown only the two questions). On the 222 children that need only new parameter values, ABCAgent matches the neural agent (62.6% against 61.7%). On the 142 that need changed logic it trails (50.0% against 57.7%, p = 0.043), so it does not yet generalize consistently when the logic changes. About half of that gap comes from rebinding being accepted for logic-change children by a check that judges plausibility rather than correctness. When the program is revised instead, it is on par (51.0% against 54.2%). We believe the main bottleneck is this check on open-ended tasks. With a check that solves the task independently, the reuse path is 99.0% correct on GSM-Symbolic. A further limit is in the variants: in a research question, varying one fact alone often has no real-world referent or breaks the question's other premises, so such a variant offers no reuse to anticipate (examples in our reply to WMGR).

Q3 (efficiency under different reuse frequencies). Figure 3 of the submission is this analysis: cost efficiency against the share of tasks solved by reuse, the number of loop iterations, and the number of evaluation passes. Appendix A also states the break-even points. In the revision, costs are measured on our API meter, which includes all overhead (the GSM-Symbolic runs predate it), and Table 1 reports authoring and inference separately, which makes the dependence on reuse explicit.

  • Inference and authoring. A task served by a stored program is 41× as cost-efficient as the neural agent on GSM-Symbolic ($0.0036 against $0.149), and about 700× on augmented GAIA ($0.004 against $2.74). Authoring a program is 0.61× and 0.37× as cost-efficient as the neural agent solving one task.
  • Break-even. A program pays back its authoring after about 0.6 reuses on GSM-Symbolic and 1.7 on augmented GAIA. GSM-Symbolic reuses each program 12.5 times (6.97× per task overall), and augmented GAIA 0.25 times (0.47×).
  • τ²-bench. Authoring pays back after about 3.0 (telecom) and 4.9 (airline) evaluation passes, and each pass after that is 12.93× and 5.66× as cost-efficient as the stock agent.
  • WorkArena-CF. Cost efficiency rises from 0.75× at 10 records to 1.18× at 30 (Figure 3b).

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts, particularly the new generalization analysis and the more comprehensive efficiency evaluation, which have addressed several of my concerns. My main remaining concern is the limited methodological novelty, especially the lack of direct comparisons with closely related programmatic agent methods. Nevertheless, the rebuttal has positively influenced my assessment. I would have liked to raise my rating by 0.5 points, but since this option is unavailable, I will maintain my original rating of weak rejection.

Reviewer_WMGROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes ABCAGENT that separates agent design, producing a behavior specification, from execution. Agent behavior is represented as a symbolic program — ordinary code with embedded calls to neural functions authored and revised by an FM Designer and run deterministically by a "Symbolic Executor." Programs declare their input parameters up front so they can later be re-bound to new task instances rather than regenerated. When ABCAGENT meets a new task, it checks whether a program already exists for that task family; if so, it substitutes the new task's parameter values into the stored program (sometimes via a lightweight FM call to parse the new values out of natural language) rather than re-invoking the FM Designer from scratch. If this substitution attempt fails, the FM Designer revises the program.

Critical strengths

[+] The design of execution split and the variant-check as a testing idea are an effective way to gate against programs whose generality is merely declared but not implemented and the authors are thorough about the ablations.

[+] In the limitations (appendix), the authors are candid about run-to-run variance, small sample sizes, contamination risk, single model generalizatoin concern, and a break-even analysis of when reuse actually pays for itself.

Critical weaknesses

[-] My concern (in line with the authors discussioin in the appendix) is around Augmented GAIA evaluation. The program solving a sibling task it hasn't seen yet is 0.332, which is not too different from the 0.302 control rate where program authored just once for a single instance and then one-shot rebound, i.e., without the benefit of the iterative revision/evolution the full system performs as it sees more family members.

[-] I'm also curious how "attempt failure" is actually detected for open-ended, generative tasks. There is brief commentary that 'an LLM-based critic detects an issue with the program', but two things are not very clear to me. First, how are "issues" defined? Second, how do you ahead of time, if something is going to be execution fail or would need the critic?

Questions for authors

Please see the questions in the weakness section regarding generative and also discussion on Augmented GAIA.

1 comment

We thank the reviewer for the careful reading and the two questions.

Augmented GAIA forward transfer. As Section 4 states, program reuse on augmented GAIA improves backward compatibility (0.469 against 0.305, p = 3.5e-5, Table 3) but forward generalization only nominally, because a forward task may add requirements its program has not seen. To make this precise, we classified each augmented-GAIA child by what its parent's program would need, using a model that sees only the two questions: 222 children need new parameter values only, 142 changed logic and 11 a new program. On parameter changes ABCAgent matches the neural agent (62.6% against 61.7%, p = 0.86); on logic changes it trails (50.0% against 57.7%, p = 0.043). About half of that gap comes from rebinding being accepted for 25 logic-change children whose answers were wrong in 11 cases; when the program is revised instead, the two are on par (51.0% against 54.2%). Hence reuse helps where variants share the parameterized structure, the first form of reuse in our general response, and the open-ended check is the bottleneck elsewhere.

A second limit lies in the variants themselves: on open-ended tasks, it is hard to tell in advance which parts of a question can vary, and how. Where a task queries a fixed source along an explicit axis, rebinding the parent's program works; children that change the reaction number in a Michaelis–Menten spreadsheet task, or which top-k lists a Box Office Mojo question compares, are all served this way. A research question instead chains facts that pin each other down: one parent asks for an AI-regulation paper submitted to arXiv in June 2022 whose figure has three labeled axes. Moving the month alone need not identify another such paper, and that child's label is the parent's answer. Moving "fifth section" to "fourth" in a question about amendments to federal rules leaves the labeling runs in disagreement, and both systems fail. No program can anticipate variants like these; rebinding serves none or one of the three children in such families. We added this discussion to Appendix A.

How failure is detected. Nothing is predicted ahead of time; every program is executed, and the outcome decides. In the authoring loop (Section 2), an execution fails on an exception, a timeout or a missing answer. Every completed execution is then reviewed by the critic: a separate FM call that sees the task, the program and its trace. The critic re-derives the answer (GAIA, GSM-Symbolic), inspects the live page (WorkArena) or judges whether the policy can complete the task (τ²-bench), and an "issue" is a disagreement it can support with evidence; authoring stops after at most 3 rounds. Before a program is stored, the variant check executes it on variants of the task; on WorkArena-CF, programs stored without it write no record correctly when replayed (Table 10). On reuse, the rebound program's answer is accepted only if a check passes. On GSM-Symbolic this check solves the instance independently and requires agreement: 99.0% of the 4,631 accepted answers are correct, and it rejected 133 correct ones (recall 97.2%). On GAIA, where answers are open-ended, the check judges plausibility only and accepted a wrong answer in 29 of 76 cases. When rebinding fails, the reason is usually legible in the program (Appendix G, Figure 6).

To measure the GAIA side directly, we replayed the reuse path (rebinding, execution, check) on 303 children unseen during authoring and scored every answer against its label. The path rejected 201 of the 235 children it would have answered wrongly (86%) and kept 59 of the 68 correct answers (87%); 63% of accepted answers are correct. Most rejections happen before the check, when rebinding finds no values or the rebound program fails; on children that need changed logic, 83 of 112 return to authoring this way.

Reviewer_Qgg8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ABCAGENT, which represents agent behavior as an executable program and separates program design from execution. An FM Designer writes or updates the program, while a symbolic executor runs it. For related tasks, an existing program can be reused by rebinding the parameters instead of generating new behavior from scratch. The method is evaluated on GAIA, WorkArena, GSM-Symbolic, τ2-bench, and two augmented benchmarks. The clearest gains appear on structured or recurring tasks such as GSM-Symbolic and τ2-telecom, where program reuse might reduce cost substantially. Additionally, the ablation results also indicate that reuse and rebinding could account for most of the efficiency gain, while the variant check appears important for avoiding incorrect program reuse. The evaluation also covers several different agent settings, and the ablation study could provide useful evidence about where the gains come from.

Critical strengths

The main idea seems sound and well motivated from a practical perspective: stable control logic can be taken out of repeated LLM reasoning and made explicit in a program. Intuitively, this should be most beneficial when tasks are repeated often. Compared to this, the paper evaluates different agent settings and performs ablation studies, which are probably best suited to understanding where the performance improvements come from. The results seem meaningful and strong.

Critical weaknesses

W1 — The efficiency evidence is less convincing than the main presentation suggests. The reported speedups may exclude part of the system overhead. Appendix B indicates that some variant-check calls might not be timed on GAIA and WorkArena-CF, and script execution outside model calls could also be excluded. Under an end-to-end wall-clock measure, ABCAGENT appears actually slower than the neural agent on GAIA and augmented GAIA. Since efficiency represents one of the main claims of the paper, the current results may not fully support a broad claim of faster execution.

W2 — The experiments do not clearly establish that ABCAGENT outperforms other program-based agent methods. Most of the direct comparisons seem to be with neural agents. However, the related work already lists several methods that use generated programs, reusable skills, or executable workflows, and comparison with those methods will be more convincing in establishing the advantages of ABCAGENT. Without direct comparison with at least one such method, it would be hard to tease out the specific benefits of ABCAGENT's design from more general benefits of programmatic execution. This also makes it difficult to assess the empirical novelty.

W3 — The WorkArena L1 result is over-interpreted. ABCAGENT achieves a success rate of ~62%, compared with ~84% for the neural agent, with a gap of over 20 percentage points. Although the authors admit that this benchmark may be underpowered, it is still not sufficient to show that the two methods have similar accuracies. It would be more reasonable to consider this result as inconclusive.

W4 — The augmented GAIA evaluation has to rely on automatically generated ground truth, with answers derived from multiple Co-Sight runs and majority voting. Indeed, the authors themselves admit that the labels are not entirely reliable. Since this benchmark tests program reuse and generalization, small performance differences may be hard to interpret without more rigorous human verification of the ground-truth answers.

Questions for authors

Q1. Could the authors report end-to-end wall-clock latency on the main benchmarks, including variant checking, program execution, and other related overhead?

Q2. How does ABCAGENT compare with other closely related program-based or reusable-skill agents? A direct comparison would make its specific advantage clearer.

Q3. Given the large difference in success rate and limited statistical power, how should the authors interpret the WorkArena L1 results?

Q4. How much of the augmented GAIA has been manually verified, and how sensitive are the reported results to errors in the automatically generated ground-truth labels?

2 comments

We thank the reviewer for the positive assessment and the specific questions.

W1/Q1 (end-to-end wall clock). Appendix B of the submission reports end-to-end wall clock for GAIA (0.46×) and augmented GAIA (0.64×) and lists what Table 1's speedup excludes. For all main benchmarks (neural time ÷ ABCAgent time):

Table 1 speedup (completed model calls)wall clock
GAIA / augmented GAIA0.85× / 1.33×0.46× / 0.64×
WorkArena L1 / CF0.36× / 0.94×0.32× / 0.81×
GSM-Symbolic5.21×2.6×
τ²-telecom / airline (agent time)9.50× / 4.08×9.50× / 4.08×

On τ²-bench we report the agent's own time, as Table 1 does (Appendix B). The benchmark's simulated user is part of neither agent. The revised Table 1 separates the time of authoring a program from the time of serving a task with it. By wall clock, the overhead on GAIA and augmented GAIA is all authoring: authoring a program is 0.46× (GAIA) and 0.51× (augmented GAIA) as fast as the neural agent solving one task. Serving a task with a stored program is fast:

  • about 5 s per rebound augmented-GAIA task, against 18 minutes for the neural agent (~200× by wall clock);
  • 3.6 s per GSM-Symbolic instance, against 50 s (14×);
  • 8 s per WorkArena-CF record, against 81 s (~10×).

On augmented GAIA, where 20.1% of the tasks are served this way, the authoring time dominates; on GSM-Symbolic (92.6%) and τ²-bench (every evaluation episode), the inference time does. This is the reuse dependence that Figure 3 and Appendix A describe.

W2/Q2 (program-based methods). The components that set ABCAgent apart from other program-generating agents are rebinding and the variant check, and Table 2 ablates both inside one harness. Without reuse and rebinding, ABCAgent writes a program per task, much like an agent that acts through code; this setting is 0.96× as cost-efficient as the neural agent on GSM-Symbolic, against 11.91× with rebinding. When ABCAgent's executor replays programs stored without the variant check on WorkArena-CF, they write no record correctly (Table 10). The neural baseline, Claude Code, itself acts by writing and executing code (CodeAct-style), and Section 5 discusses CodeAct, skill induction and the concurrent LLM-as-Code. To relate this to LATM, which writes a tool from a few examples and applies it without further checks, we froze ABCAgent's GSM-Symbolic program after at most 4 instances and applied it to the later instances with the same binding call, but without the check or the fallback: it solves 95.4% of 4,637 instances, against 99.1% for full ABCAgent (+3.6 points, family-bootstrap 95% CI [+1.8, +5.8]). Since the fallback authoring sees the first instance's reference solution (Appendix B), this is an upper bound on what the check and the fallback add. For the final version, we will add LATM and a programmatic skill-induction agent (ASI) in our harness, with the same model and meter; their published implementations target narrower settings (few-shot reasoning; web navigation), so not every benchmark here admits a faithful version of both.

W3/Q3 (WorkArena L1). We agree and have revised Section 3 accordingly: 62.4% against 84.9% (p = 0.041) is a loss at α = 0.05, and WorkArena L1 is no longer listed with the benchmarks where the systems are indistinguishable. Section 4 and Appendix G of the submission analyze its cause: most failures leave the browser on a page other than the one the validator reads; the 9 ordering tasks account for most of the gap (48% against 100%). Since these tasks are one-offs, this is also where Appendix A places ABCAgent below its break-even point.

W4/Q4 (augmented GAIA labels). The labels are not Co-Sight's alone: as Appendix A notes, the authors validated them where errors were found, before any of the reported runs. In total, 66 of the 378 child tasks (17%) carry a label or question corrected by the authors. We aligned the format of all labels by hand, which changed 25, and re-examined every child on which two of ABCAgent's runs converged on an answer other than the label. That check changed 38 labels and the wording of 6 ambiguous questions, and removed 4 tasks. Because both systems are scored against the same labels, the comparison is stable across subsets of differing label reliability (ABCAgent − neural, McNemar p):

subsetndiffp
all children378−2.40.28
all three Co-Sight runs agree179−4.50.15
never edited312−2.60.29

A label error can change the paired comparison only on the 125 children where exactly one system passes, or both fail with different answers. We will audit all of them, blind to which system produced each answer, and add the result to the final version; this needs more annotation time than the discussion period allows.

Thank you for the clarification. My concerns have been largely addressed. I maintain my positive assessment and will keep my score unchanged. I encourage the authors to update the manuscript accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.