acceptodds
Under review as a conference paper at ICLR 2027

Generalization Beyond Coding: An Empirical Comparison of Coding Agents and Purpose-Built Agents

Abstract

Recent advances in large language models (LLMs) have enabled both flexible coding agents and specialized, purpose-built agents to tackle complex tasks. General-purpose coding agents such as Codex show strong adaptability, while purpose-built agents rely on domain-specific structures to solve specialized problems. This prompts a key question: Are coding agents now advanced enough to make purpose-built engineering obsolete? This paper presents a comprehensive empirical comparison between Codex and comparable state-of-the-art purpose-built LLM agents across 20 non-coding benchmarks spanning finance, clinical reasoning, data analysis, information retrieval, and scientific research. For each benchmark, an AI-assisted procedure finds the comparable state-of-the-art purpose-built agent with released code, verifies its reported score, and re-runs it with the same language model as Codex; both agents are scored on the same instances by the same grader. Using the same underlying model and evaluation methods, Codex significantly outperforms on 5 benchmarks, shows no significant difference on 13, and underperforms on only 2. Many of the advantages previously attributed to purpose-built agents diminish when controlling for the backbone model and evaluation protocols. In certain cases, the additional complexity introduced by staged data processing even results in poorer performance compared to Codex’s direct, end-to-end approach. Where purpose-built agents do win, much of their advantage is portable: their domain tools, handed to Codex, carry much of the benefit, and their decisive workflow steps encode procedural knowledge that Codex can execute but rarely finds on its own; once given them, it matches the purpose-built agents on most of the selected instances it had lost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Coding Agent, Agent evaluation, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MKgDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares a general-purpose coding agent, Codex, with purpose-built agents across 20 non-coding benchmarks covering scientific, medical, financial, retrieval, and data-analysis tasks. The main comparison uses a shared primary backbone, identical evaluation instances, and common graders, covering 4,649 instances. Codex significantly outperforms the specialized systems on five benchmarks, shows no significant difference on thirteen, and underperforms on two. The authors additionally analyze execution traces and investigate whether providing specialized tools or workflow steps helps Codex recover failures.

Critical strengths
  1. Interesting research question. Determining when specialized agent engineering provides value over a strong general-purpose agent is relevant to both research and deployment. Re-running released systems across 20 benchmarks, constructing adapters, and collecting paired execution traces constitute useful empirical work.

  2. Careful attention to evaluation artifacts. Shared primary backbones, paired instances, common graders, leakage checks, and investigations of answer-format issues improve substantially over comparisons based solely on published scores. The documentation of implementation changes and sensitivity analyses is also valuable.

Critical weaknesses
  1. The comparisons do not fully isolate agent design. Although the primary backbone is shared, auxiliary models, tools, and information access still differ. Performance gaps may therefore reflect these resources rather than the value of specialized workflows.

  2. Modifications to specialized agents introduce additional uncertainty. Some systems use incomplete released implementations or modified operating conditions, including default sampling settings. These changes are disclosed, but their effects on performance are not sufficiently quantified.

  3. The proposed explanations lack strong intervention evidence. Trace analysis identifies plausible reasons for success or failure, but does not establish their contribution. Tool- and step-transfer experiments use selected failure cases, and successful reruns cannot distinguish intervention benefits from ordinary run-to-run variation without matched controls.

  4. The conclusions have limited generality and statistical support. The main comparison uses one coding agent and one primary backbone. Several benchmark subsets are small, and the thirteen non-significant differences do not establish equivalence between general-purpose and specialized agents.

Questions for authors

See weakness

6 comments

We thank the reviewer for recognizing the value of our research question and the rigor of our evaluation. To address the concerns, we clarify our study’s scope (Weakness 1), assess the impact of implementation changes (Weakness 2), add unchanged-Codex reruns and matched-inference-budget controls (Weakness 3), and evaluate a second coding agent, Claude Code with Opus 5.5, while expanding the Codex evaluation samples (Weakness 4).

Response to Weakness 1

We thank the reviewer for raising this important point. We explicitly consider the different components of agent design; our analysis proceeds in three stages. First, holding the primary backbone constant, we compare the system-level performance of purpose-built agents and a strong general coding agent, treating each agent’s tools, information access, auxiliary models, and workflows together as a complete harness system.

Second, we go beyond aggregate scores to investigate which components contribute to the observed differences. We group benchmarks by information access and analyze paired execution traces at the instance level. Of 275 purpose-built-only wins, 49 involve unique data or tools, 66 explicit workflow steps, 103 different uses of the same information, and 57 format, setup, or reference issues. Conversely, 174 of 388 Codex-only wins involve information lost within purpose-built pipelines. This distinguishes advantages associated with resources(tools, auxiliary models) from those associated with workflow steps and information use. [Section 4.5; Appendix I.3.]

Finally, we test whether the resources and steps identified in the traces provide practical value when transferred to Codex. These include RDKit tools from S1-NexusAgent (ChemBench), remote-sensing tools from Earth-Agent (EarthBench), ARK’s graph-query tools (STaRK-PrimeKG), GSEM’s case memory (MedAgentsBench), and RARG’s document ranker (BRIGHT), as well as workflow steps from MoCA, EnSI-RAG, SearchOS, and AgentSPEX[Section 4.5.]. Against unchanged-Codex controls on the same targets, resource transfers recover 61 of 106 cases versus 33 without. On the expanded set of 43 step-transfer targets, they recover 32 instances versus 10 with unchanged Codex on a single run, with gains persisting under approximately matched inference budgets [Added in the revision: Figure 3; Appendix I.3, Tables 17 and 18.].

Together, this progression compares complete systems, investigates the design and resource differences behind their performance gaps, and tests the value of reusable components. We therefore do not attribute every gap to specialized workflows; distinguishing resource-related from procedure-related advantages is part of our study.

Response to Weakness 2

We thank the reviewer for raising this concern. More broadly, reproducing published scientific work remains difficult even with AI assistance, as highlighted by VERITAS [1] and ReplicationBench [2]. We therefore worked to reproduce each agent as faithfully as possible from its released code and published description, documenting the changes that were necessary. We agree that documenting these changes is not enough; we need to assess their impact on the comparison. We distinguish three cases. First, some adaptations can be measured directly: our checks for ASTRA, AgentSPEX, and REMem-I do not reverse the tested comparisons. Second, this is an interface limitation, not a deliberate change: Newer models like GPT-5.6-sol and Claude Opus 5.5 do not support user-set temperatures at the reasoning settings used here, so we use their defaults. Third, RARG+ and MoCA include components that were not released; we follow the original papers’ results without those components, and both systems still satisfy our comparable state-of-the-art agent criterion. Thus, the adaptations are either empirically checked, required for deployment on the evaluated backbones, or supported by published ablations; none undermines our main comparative findings.

ASTRA (SSTQA). ASTRA’s released pipeline returns a textual answer and does not call its answer selector, which is released as a separate script, so we scored the textual answer. With the selector added (its released prompt, on GPT-5.6-luna), ASTRA rises from 73.8 to 77.5, and Codex’s lead falls from +11.5 to +7.9, still significant (p = .003). [Added in the revision: Appendix I.1.]

AgentSPEX (ELAIPBench). Both agents run at medium reasoning effort, while AgentSPEX’s paper uses high effort. At high effort, AgentSPEX answers 190 instead of 184 of 403 instances correctly, no significant difference (p = .41). [Appendix I.3.]

REMem-I (Test of Time). Its fact extraction makes about 300 model calls per question, so we ran extraction on the smaller GPT-5.6-luna and answering on the backbone; the paper runs both on GPT-4.1-mini. With extraction also on the backbone, 60 of the first 70 questions are correct either way. [Added in the revision: Appendix I.1.]

RARG (BRIGHT) and MoCA* (FinanceMath, MultiHiertt). RARG+’s entry-point module and MoCA’s baseline proposer were not released. We therefore follow the original papers’ reported results without those components (RARG: 51.8; MoCA on MultiHiertt: 67.2) and use the released implementation where applicable. Even without the missing components, both remain the strongest comparable agents in their published comparisons and meet our comparable-SOTA selection criterion. [Appendices H and I.1.]

Taken together, our checks suggest that these adaptations do not materially affect our main comparative conclusions.

[1] H. Liu, F. A. Tjiaranata, C. Tan. VERITAS: Towards a General-Purpose Replication Tool for Scientific Research. arXiv preprint arXiv:2607.02931, 2026. doi:10.48550/arXiv.2607.02931.

[2] C. Ye, S. Yuan, S. Cooray, S. Dillmann, I. L. V. Roque, D. Baron, P. Frank, S. Martin-Alvarez, N. Koblischke, F. J. Qu, D. Yang, R. Wechsler, I. Ciuca. ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers? arXiv preprint arXiv:2510.24591, 2025. doi:10.48550/arXiv.2510.24591.

Response to Weakness 3

We agree that trace analysis alone cannot establish the contribution of individual components. We use it to identify candidate resources and workflow steps. And we strengthen intervention evidence of our existing resource- and step-transfer experiments by adding unchanged-Codex rerun experiments and comparing step transfers against an approximately matched inference budget. With task prompts and core execution settings held fixed, these paired comparisons test whether the transferred components improve Codex beyond ordinary rerun variation or additional inference alone.

Resource transfer. Across all 106 eligible purpose-built-agent wins, providing Codex with the corresponding resources recovers 61 targets, compared with 33 for unchanged Codex on the same targets (p < .001). This provides controlled evidence that transferring resources from purpose-built agents can improve Codex beyond ordinary rerun variation. These resources require no additional backbone calls, so both conditions have the same inference budget. [Added in the revision: Figure 3; Appendix I.3, Tables 16.]

Step transfer. Transferring purpose-built workflow steps improves Codex’s recovery of purpose-built-agent wins: 32 of 43 targets recover with the packages versus 10 with unchanged Codex (p < .001). The gain persists at approximately matched inference budgets: on 24 targets, the packages recover 17 versus 6 with unchanged Codex (p < .001). For FinanceMath, MultiHiertt, and ELAIPBench, package inference was 4.0×, 5.4×, and 1.8× that of a plain run, respectively; unchanged Codex received 4, 5, and 3 independent attempts, respectively, combined by majority vote under a rule fixed in advance. This rules out additional inference alone as the explanation for the gain.[Added in the revision: Figure 3; Appendix I.3, Tables 17 and 18.]

Together, these controlled comparisons provide the intervention evidence the reviewer requested: both resource and step transfers improve Codex over unchanged-Codex controls, and the step-transfer gains persist under approximately matched inference budgets.

Reviewer_ExtzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper comprehensively compares general-purpose coding agents against domain-specific agents with the same backbone (GPT-5.6-sol) across four broad categories of benchmarks (20 benchmarks in total). The work selects one representative agent design for each benchmark and performs the comparison.

The contribution is timely given the remarkable advances in coding agents in recent months. The evaluation is rigorous and methodical. Potential data leakage is well guarded against, and remedial measures in cases of unintended web searches are described sufficiently. Thus, the comparison is overall fair. The work concludes that Codex mostly outperforms purpose-built agents, and in cases where it does not, the advantage is attributed to procedural knowledge that Codex could also benefit from once it obtains that knowledge.

Critical strengths
  • The evaluation includes a much broader range of tasks than the closest work by Bandel et al. (Feb 2026), which covers only six benchmarks and evaluates general-purpose agent designs. The category covering science and medicine is unique to this paper. Importantly, for certain domains like science, comparing representative agents in that domain against general-purpose agents may be more insightful than comparing ReAct and Claude Code (as in Bandel et al.).

  • The five research questions are very practical for future engineering decisions. They also point to future research on a taxonomy of which types of procedural knowledge from purpose-built agents are most useful.

  • Implementation details, including how the benchmarks are adapted, are clearly outlined, which is great for reproducibility.

Critical weaknesses
  • The work could be greatly enhanced with a more in-depth comparison in terms of costs. Currently, the paper discusses costs in Appendix I.1, Figure 4, and Table 11, which only show the costs for questions that are correctly answered by both sides. Showing the costs for cases where one method fails but the other does not would be helpful.

  • Certain benchmarks, including InsightBench, SSTQA, etc., are judged by older models. For example, InsightBench is evaluated using GPT-4o. Using a less capable judge to evaluate a more capable backbone defeats the purpose of the evaluation. While the judge affects Codex and purpose-built agents simultaneously, the authors could reconsider whether staying faithful to the respective benchmark's evaluation protocol is more important than potentially gauging performance more accurately with more capable judges.

Questions for authors

As above

3 comments

We thank the reviewer for recognizing the breadth, practical value, and reproducibility of our evaluation. We appreciate the constructive suggestions on cost analysis and LLM-based evaluation, which we address with additional experiments below.

Response to Weakness 1

We thank the reviewer for this suggestion. We have conducted a more detailed cost analysis across all 20 benchmarks, including instances where only one agent succeeds, rather than restricting the comparison to questions both agents answer correctly. We find that Codex's lower cost is not limited to instances both agents solve: purpose-built agents cost more on average in every outcome group, including instances only they solve. For the 16 binary-scored benchmarks, we group instances into four outcomes:

OutcomeInstancesPurpose-built cost ÷ Codex cost
Both are correct2,8691.71×
Only Codex is correct3101.82×
Only the purpose-built agent is correct1911.33×
Both are incorrect1,0181.59×

Notably, on the 310 instances where Codex succeeds and the purpose-built agent fails, the latter costs 1.82× as much. The same pattern holds for the four continuous-score benchmarks, where purpose-built agents cost 5.7× as much when Codex performs better, 3.8× when they perform better, and 1.2× on ties. More broadly, purpose-built agents spend an average of USD 0.44 per correct answer versus USD 0.26 for Codex, and USD 0.55 per incorrect answer versus USD 0.33. Thus, purpose-built agents are more expensive in both cases, while incorrect answers also cost more than correct ones on average for both codex and purpose-built agents.

At the individual agent level, we further find that higher cost does not necessarily translate into better performance. The two purpose-built agents with significant performance advantages, Earth-Agent on EarthBench and EnSI-RAG on OOLONG, are actually cheaper than Codex (0.62× and 0.46×, respectively). Earth-Agent benefits from using appropriate computational tools, while EnSI-RAG reduces costs by delegating tasks to a smaller auxiliary model. In contrast, the two more expensive agents—SearchOS (18×), and FrontierAgent (5×)—show no significant performance advantage over Codex.

Overall, effective specialization depends on how resources and workflows are used, rather than simply spending more. We have added the detailed cost analysis to Appendix I.1, Table 13, and Section 4.3.

Response to Weakness 2

We thank the reviewer for this suggestion. We use newer models as judge and find that our results are robust to stronger LLM judges: across all four LLM-judged benchmarks, neither the direction nor the statistical significance of the performance differences changes. To be specific, we re-scored the existing answers with GPT-5.6-Terra, retaining each benchmark’s official prompt and scoring rule. For InsightBench, the official G-Eval metric requires token probabilities, which GPT-5.6-Terra does not provide. We therefore additionally re-judged InsightBench with GPT-5.4, the newest model available to us that supports these probabilities, to preserve the official protocol. The results are shown below:

Benchmark (purpose-built agent)nOriginal judgeΔ [95% CI], pNew judgeΔ [95% CI], p
SSTQA (ASTRA)191GPT-5+11.5 [+4.7, +19.5], <.001GPT-5.6-Terra+11.0 [+4.2, +19.0], <.001
CRAG (CacheRAG)578Claude Sonnet 4.5+5.0 [+0.9, +9.3], .02GPT-5.6-Terra+4.7 [+0.5, +8.8], .04
FrontierScience (FrontierAgent)58GPT-5, high effort+6.9 [−3.4, +17.5], .34GPT-5.6-Terra+5.2 [−5.3, +16.9], .55
InsightBench (DataSTORM)25GPT-4o, G-Eval−3.0 [−7.4, +3.0], .42GPT-5.4, G-Eval−0.6 [−4.6, +4.1], .82

Δ is Codex minus the purpose-built agent in points, with the intervals and tests of Table 1. The original and new judges also agree item by item: they give the same verdict on 96–98% of the judged answers on SSTQA and CRAG and on 81–90% on FrontierScience, where the differences are mostly answers near the pass mark, and on InsightBench GPT-5.4’s per-dataset scores correlate with GPT-4o’s at 0.74–0.93.

We therefore retain the original judges for the primary comparison to preserve consistency with each benchmark’s established protocol, while reporting stronger-judge results as a robustness check; the conclusions remain unchanged. [Added in the revision: Section 4.2; Appendix I.2, Table 15.]

Thank you for the detailed response and the clarifications. All my concerns are fully addressed. I decide to keep my initial rating.

We sincerely thank you for your thoughtful comments, as well as your encouraging rating. Your suggestions are valuable in helping us improve our paper. We appreciate your time and willingness to reply.

Reviewer_9nsDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper compares Codex CLI with 17 purpose-built agents across 20 non-coding benchmarks and 4,649 paired instances. The authors rerun the specialized agents with the same main backbone as Codex, use shared instances and graders, and audit protocol differences, answer leakage, and output extraction. They report five significant Codex wins, thirteen nonsignificant differences, and two significant losses. Paired trace analysis examines information loss within specialized pipelines, while additional experiments test transferring domain tools and workflow steps to Codex.

Critical strengths
  • The paper includes a substantial paired comparison that improves on published-score comparisons. Rerunning specialized agents on shared instances, graders, and a common main backbone across twenty benchmarks is a meaningful empirical contribution. Paired tests, confidence intervals, and documented adaptations make the differences substantially more interpretable than comparisons between independently published scores (Table 1; Appendices D and I.1).

  • The paper examines both agents’ traces and identifies specific extraction, filtering, and selection failures. It associates 174 of 388 Codex-only wins with information lost inside specialized pipelines, with a stricter labeling sensitivity analysis. It also demonstrates that leakage and answer-location handling can change comparative conclusions (Sections 4.2-4.6; Appendix I). These findings give the study value beyond reporting which agent scores higher.

Critical weaknesses
  • The category-level conclusion exceeds the tested coverage. All coding-agent results use one Codex version, one backbone, and one reasoning-effort setting. Benchmark breadth establishes task coverage, but does not establish robustness across coding agents or backbones. Appendix C explicitly limits “Codex” to this configuration, whereas the conclusion makes broader recommendations about when specialized solutions remain necessary. This limits the generality of the paper’s central engineering recommendation. The thirteen nonsignificant comparisons also establish neither equivalence nor noninferiority.

  • The portability experiments do not isolate the benefit of the transferred components. The reported recoveries (61/106 with resources and 22/27 with workflow steps) come from rerunning selected cases that Codex previously lost, without an unchanged-Codex rerun control. Some recoveries could therefore reflect stochastic variation. Furthermore, step transfer executes the specialized agent’s prompts and procedure through separate model calls, which limits what it establishes about Codex incorporating reusable guidance into its own workflow (Appendix I.3). These issues weaken a headline explanatory claim, although they do not invalidate the main paired comparison.

Questions for authors
  • What selection rule produced the 27 step-transfer targets, and how large was the eligible pool? Were they selected before observing any transfer outcomes?

  • On the same targets, what recovery rate does unchanged Codex achieve under matched sampling and inference budgets? The improvement over this control would materially affect my assessment of the portability claim.

  • Does the comparative pattern persist with a second coding-agent or backbone configuration on a small subset spanning the three information groups? This would help distinguish configuration-specific findings from broader design conclusions.

5 comments

We thank the reviewer for recognizing the rigor of our empirical evaluation and the broader insights our analysis provides beyond aggregate performance comparisons. We appreciate the concerns about generality and whether the transfer gains exceed ordinary rerun variation. To address them, we evaluated Claude Code on Opus 5.5 (Weakness 1, Question 3), added unchanged-Codex controls at original and matched inference budgets (Weakness 2, Question 2), expanded the step-transfer evaluation and clarify target selection (Question 1).

Response to Weakness 1 and Question 3:

To test whether our findings are specific to Codex on GPT-5.6-sol, we repeated the evaluation with Claude Code (v2.1.319) and the purpose-built agents on Claude Opus 5.5 at medium reasoning effort with all other same settings following Table 1. We attempted this second-configuration evaluation on 12 benchmarks: five with significant Codex wins, two with significant Codex losses, and five with no significant difference in the original configuration. Three could not be compared because Opus 5.5 refused to respond: STaRK-PrimeKG and Biomni-Eval1 involved biology-related requests, while CRAG’s purpose-built agent uses prompts that request written reasoning, which also triggered a refusal.

Information groupBenchmarknCodex, GPT-5.6-sol (Table 1)Claude Code, Opus 5.5
Converts or filters inputSSTQA (zh)191+11.5 [+4.7, +19.5]+5.2 [+1.0, +9.4]
Test of Time280+6.8 [+2.9, +11.0]+4.3 [+1.8, +6.8]
ELAIPBench403−0.5 [−5.1, +4.0]+0.7 [−2.5, +3.7]
Same informationMultiHiertt261+1.5 [−1.5, +4.6]−1.1 [−3.1, +0.8]
FinanceMath2000.0 [−4.5, +4.5]−1.5 [−5.5, +2.0]
GISA50−6.1 [−16.5, +3.5]−6.5 [−16.1, +1.7]
OOLONG79−7.7 [−13.4, −2.4]−26.7 [−37.9, −15.7]
Data or tools Codex lacksBRIGHT106+2.1 [−1.6, +5.8]+1.9 [−2.7, +6.8]
EarthBench54−20.4 [−35.2, −5.6]−20.4 [−35.2, −5.6]

Δ is the score difference (general minus purpose-built agent) in points, with paired-bootstrap 95% CIs. In both configurations, the general agent leads significantly on SSTQA and Test of Time, the purpose-built agent on OOLONG and EarthBench, and the remaining five comparisons show no significant difference. Thus, the significance pattern is unchanged despite differences in effect size. The identical EarthBench differences are coincidental (64.8 vs. 85.2 on Opus 5.5; 61.1 vs. 81.5 in Table 1). In conclusion, all the other nine yielded comparable results, with the same statistical significance pattern as in the original configuration. [Added in the revision: Section 4.3; Appendix I.7, Table 20.]

Non-significant comparisons. We agree that non-significant comparisons establish neither equivalence nor non-inferiority. We use “no significant difference” in paper to faithfully report what the statistical tests show, not to suggest that the two agents are equivalent. More importantly, our comparison goes beyond aggregate scores: we conduct instance-level trace analysis to understand where Codex and purpose-built agents differ and why. For example, on FinanceMath, both agents score 79.0, yet each solves 10 instances the other misses; across the 13 benchmarks, Codex alone wins 206 instances and the purpose-built agents alone win 181. Across all 20 benchmarks, we analyzed the traces of every instance solved by only one agent, tested the factors identified in those traces by transferring resources and workflow steps to Codex with an unchanged-Codex control (Question2), and compared model costs. Our design conclusions rest on this evidence about where the agents’ advantages come from, whether they transfer, and at what cost—not on treating non-significance as equivalence. Most step-transfer targets, for instance, are purpose-built-agent wins on benchmarks with no significant aggregate difference (FinanceMath, MultiHiertt, GISA, ELAIPBench).[Table 11; Section 4.5 and Appendices I.3–I.4.]

Response to Question 1

We selected step-transfer targets in two stages. First, we identified purpose-built agents with explicit workflow steps that Codex rarely performs on its own, such as MoCA's recomputation from raw table cells, EnSI-RAG's record-by-record counting, and SearchOS's query cleaning and answer reformatting. Across these benchmarks, the purpose-built agents solved 48 instances that Codex missed. Then we selected 27 of them based on trace analysis: the purpose-built agent used the step to obtain the correct answer, while Codex made the type of error that the step was designed to prevent. This selection was made before any transfer experiments. To check whether the results depended on selecting these 27 instances, we also evaluated all 48 wins. With the transferred steps, Codex recovered 32 instances, compared with 14 for unchanged Codex. In the paired comparison, 18 instances were recovered only with the steps and none only without them (p < .001). Thus, the improvement does not depend on the original instance selection.

Response to Weakness 2 and Question 2

To distinguish transfer gains from rerun variability or additional inference, we compared both step and resource transfers against unchanged Codex on the same targets, keeping the version, model, effort, container, and task prompt identical.

Steps. On the original 27 targets, recovery improves from 7 with unchanged Codex to 22 with the transferred step package (p < .001). We further extended the candidate set with ELAIPBench since trace analysis attributes 16 of AgentSPEX’s 42 wins over Codex to its answer re-check; 10 of these 16 recover with the packaged step versus 3 with unchanged codex. Moreover, step packages add model calls; we therefore compared them against unchanged Codex at approximately matched inference budgets. Where package inference was substantially higher, unchanged Codex received independent attempts combined by majority vote, with ties resolved by the earliest attempt under a rule fixed before the runs.

BenchmarkTargetsPackage inference ÷ one plain runWith packageUnchanged Codex, one runUnchanged Codex, matched budget
FinanceMath44.0311 (4 attempts)
MultiHiertt45.4412 (5 attempts)
ELAIPBench161.81033 (3 attempts)
OOLONG91.384–
GISA101.071–

The table summarizes 43 step-transfer targets across five benchmarks, including 16 AgentSPEX answer re-check targets on ELAIPBench. With the transferred steps, Codex recovers 32 targets versus 10 for unchanged Codex on a single run. For the 24 targets with matched-budget controls (FinanceMath, MultiHiertt, and ELAIPBench), the packages recover 17 versus 6 at approximately the same inference budget (p < .001). These controls show that the gains exceed ordinary rerun variation and cannot be explained by additional inference alone. [Added in the revision: Figure 3; Appendix I.3, Tables 17 and 18.]

Interpretation of step transfer. We agree that invoking an external component does not establish that Codex has internalized its reasoning. We didn't overclaim and our results want to show that a purpose-built agent’s procedure can be packaged as a reusable component that Codex can invoke. Each package includes the original prompts, control rule, and usage condition. Codex receives it as a skill it can independently list and open, without changing the task prompt or forcing invocation. We will therefore distinguish component portability from autonomous skill selection. Selecting the right component at the right time from a library without usage conditions remains future work.

Resources. We also add unchanged-Codex controls to the resource transfer experiments. With the resource, 61 of the 106 targets recover, against 33 on the plain re-run (p < .001). The gain is largest where the resource gives information that Codex cannot otherwise obtain, such as Earth-Agent’s analysis tools on EarthBench (14 against 4 of 15). On STaRK-PrimeKG and MedAgentsBench, Codex can reach equivalent information without the resource: ARK’s graph-query tools search the same knowledge graph that Codex reads, and the medical questions mostly call on knowledge that the shared backbone gives both agents. There, the plain re-run also recovers many targets (18 of 49), and the purpose-built agent’s advantage lies mostly in how it uses that information; the resource still adds recoveries (26 against 18).[Added in the revision: Figure 3; Appendix I.3, Table 16.]

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.