- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper compares a general-purpose coding agent, Codex, with purpose-built agents across 20 non-coding benchmarks covering scientific, medical, financial, retrieval, and data-analysis tasks. The main comparison uses a shared primary backbone, identical evaluation instances, and common graders, covering 4,649 instances. Codex significantly outperforms the specialized systems on five benchmarks, shows no significant difference on thirteen, and underperforms on two. The authors additionally analyze execution traces and investigate whether providing specialized tools or workflow steps helps Codex recover failures.
Critical strengths
-
Interesting research question. Determining when specialized agent engineering provides value over a strong general-purpose agent is relevant to both research and deployment. Re-running released systems across 20 benchmarks, constructing adapters, and collecting paired execution traces constitute useful empirical work.
-
Careful attention to evaluation artifacts. Shared primary backbones, paired instances, common graders, leakage checks, and investigations of answer-format issues improve substantially over comparisons based solely on published scores. The documentation of implementation changes and sensitivity analyses is also valuable.
Critical weaknesses
-
The comparisons do not fully isolate agent design. Although the primary backbone is shared, auxiliary models, tools, and information access still differ. Performance gaps may therefore reflect these resources rather than the value of specialized workflows.
-
Modifications to specialized agents introduce additional uncertainty. Some systems use incomplete released implementations or modified operating conditions, including default sampling settings. These changes are disclosed, but their effects on performance are not sufficiently quantified.
-
The proposed explanations lack strong intervention evidence. Trace analysis identifies plausible reasons for success or failure, but does not establish their contribution. Tool- and step-transfer experiments use selected failure cases, and successful reruns cannot distinguish intervention benefits from ordinary run-to-run variation without matched controls.
-
The conclusions have limited generality and statistical support. The main comparison uses one coding agent and one primary backbone. Several benchmark subsets are small, and the thirteen non-significant differences do not establish equivalence between general-purpose and specialized agents.
Questions for authors
See weakness
6 comments
Response to Weakness 2
We thank the reviewer for raising this concern. More broadly, reproducing published scientific work remains difficult even with AI assistance, as highlighted by VERITAS [1] and ReplicationBench [2]. We therefore worked to reproduce each agent as faithfully as possible from its released code and published description, documenting the changes that were necessary. We agree that documenting these changes is not enough; we need to assess their impact on the comparison. We distinguish three cases. First, some adaptations can be measured directly: our checks for ASTRA, AgentSPEX, and REMem-I do not reverse the tested comparisons. Second, this is an interface limitation, not a deliberate change: Newer models like GPT-5.6-sol and Claude Opus 5.5 do not support user-set temperatures at the reasoning settings used here, so we use their defaults. Third, RARG+ and MoCA include components that were not released; we follow the original papers’ results without those components, and both systems still satisfy our comparable state-of-the-art agent criterion. Thus, the adaptations are either empirically checked, required for deployment on the evaluated backbones, or supported by published ablations; none undermines our main comparative findings.
ASTRA (SSTQA). ASTRA’s released pipeline returns a textual answer and does not call its answer selector, which is released as a separate script, so we scored the textual answer. With the selector added (its released prompt, on GPT-5.6-luna), ASTRA rises from 73.8 to 77.5, and Codex’s lead falls from +11.5 to +7.9, still significant (p = .003). [Added in the revision: Appendix I.1.]
AgentSPEX (ELAIPBench). Both agents run at medium reasoning effort, while AgentSPEX’s paper uses high effort. At high effort, AgentSPEX answers 190 instead of 184 of 403 instances correctly, no significant difference (p = .41). [Appendix I.3.]
REMem-I (Test of Time). Its fact extraction makes about 300 model calls per question, so we ran extraction on the smaller GPT-5.6-luna and answering on the backbone; the paper runs both on GPT-4.1-mini. With extraction also on the backbone, 60 of the first 70 questions are correct either way. [Added in the revision: Appendix I.1.]
RARG (BRIGHT) and MoCA* (FinanceMath, MultiHiertt). RARG+’s entry-point module and MoCA’s baseline proposer were not released. We therefore follow the original papers’ reported results without those components (RARG: 51.8; MoCA on MultiHiertt: 67.2) and use the released implementation where applicable. Even without the missing components, both remain the strongest comparable agents in their published comparisons and meet our comparable-SOTA selection criterion. [Appendices H and I.1.]
Taken together, our checks suggest that these adaptations do not materially affect our main comparative conclusions.
[1] H. Liu, F. A. Tjiaranata, C. Tan. VERITAS: Towards a General-Purpose Replication Tool for Scientific Research. arXiv preprint arXiv:2607.02931, 2026. doi:10.48550/arXiv.2607.02931.
[2] C. Ye, S. Yuan, S. Cooray, S. Dillmann, I. L. V. Roque, D. Baron, P. Frank, S. Martin-Alvarez, N. Koblischke, F. J. Qu, D. Yang, R. Wechsler, I. Ciuca. ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers? arXiv preprint arXiv:2510.24591, 2025. doi:10.48550/arXiv.2510.24591.
Response to Weakness 3
We agree that trace analysis alone cannot establish the contribution of individual components. We use it to identify candidate resources and workflow steps. And we strengthen intervention evidence of our existing resource- and step-transfer experiments by adding unchanged-Codex rerun experiments and comparing step transfers against an approximately matched inference budget. With task prompts and core execution settings held fixed, these paired comparisons test whether the transferred components improve Codex beyond ordinary rerun variation or additional inference alone.
Resource transfer. Across all 106 eligible purpose-built-agent wins, providing Codex with the corresponding resources recovers 61 targets, compared with 33 for unchanged Codex on the same targets (p < .001). This provides controlled evidence that transferring resources from purpose-built agents can improve Codex beyond ordinary rerun variation. These resources require no additional backbone calls, so both conditions have the same inference budget. [Added in the revision: Figure 3; Appendix I.3, Tables 16.]
Step transfer. Transferring purpose-built workflow steps improves Codex’s recovery of purpose-built-agent wins: 32 of 43 targets recover with the packages versus 10 with unchanged Codex (p < .001). The gain persists at approximately matched inference budgets: on 24 targets, the packages recover 17 versus 6 with unchanged Codex (p < .001). For FinanceMath, MultiHiertt, and ELAIPBench, package inference was 4.0×, 5.4×, and 1.8× that of a plain run, respectively; unchanged Codex received 4, 5, and 3 independent attempts, respectively, combined by majority vote under a rule fixed in advance. This rules out additional inference alone as the explanation for the gain.[Added in the revision: Figure 3; Appendix I.3, Tables 17 and 18.]
Together, these controlled comparisons provide the intervention evidence the reviewer requested: both resource and step transfers improve Codex over unchanged-Codex controls, and the step-transfer gains persist under approximately matched inference budgets.
We thank the reviewer for recognizing the value of our research question and the rigor of our evaluation. To address the concerns, we clarify our study’s scope (Weakness 1), assess the impact of implementation changes (Weakness 2), add unchanged-Codex reruns and matched-inference-budget controls (Weakness 3), and evaluate a second coding agent, Claude Code with Opus 5.5, while expanding the Codex evaluation samples (Weakness 4).
Response to Weakness 1
We thank the reviewer for raising this important point. We explicitly consider the different components of agent design; our analysis proceeds in three stages. First, holding the primary backbone constant, we compare the system-level performance of purpose-built agents and a strong general coding agent, treating each agent’s tools, information access, auxiliary models, and workflows together as a complete harness system.
Second, we go beyond aggregate scores to investigate which components contribute to the observed differences. We group benchmarks by information access and analyze paired execution traces at the instance level. Of 275 purpose-built-only wins, 49 involve unique data or tools, 66 explicit workflow steps, 103 different uses of the same information, and 57 format, setup, or reference issues. Conversely, 174 of 388 Codex-only wins involve information lost within purpose-built pipelines. This distinguishes advantages associated with resources(tools, auxiliary models) from those associated with workflow steps and information use. [Section 4.5; Appendix I.3.]
Finally, we test whether the resources and steps identified in the traces provide practical value when transferred to Codex. These include RDKit tools from S1-NexusAgent (ChemBench), remote-sensing tools from Earth-Agent (EarthBench), ARK’s graph-query tools (STaRK-PrimeKG), GSEM’s case memory (MedAgentsBench), and RARG’s document ranker (BRIGHT), as well as workflow steps from MoCA, EnSI-RAG, SearchOS, and AgentSPEX[Section 4.5.]. Against unchanged-Codex controls on the same targets, resource transfers recover 61 of 106 cases versus 33 without. On the expanded set of 43 step-transfer targets, they recover 32 instances versus 10 with unchanged Codex on a single run, with gains persisting under approximately matched inference budgets [Added in the revision: Figure 3; Appendix I.3, Tables 17 and 18.].
Together, this progression compares complete systems, investigates the design and resource differences behind their performance gaps, and tests the value of reusable components. We therefore do not attribute every gap to specialized workflows; distinguishing resource-related from procedure-related advantages is part of our study.