acceptodds
Under review as a conference paper at ICLR 2027

Athanor: An Agent-Native System for Autonomous Post-Training Research

18 views

Abstract

Post-training research is an iterative process of diagnosing model weaknesses, testing data and algorithmic recipes, evaluating outcomes, and refining subsequent experiments from evidence. We build , an open agent-native system that enables frontier generalist agents to hill-climb a raw base model on target benchmarks through long-running campaigns of training experiments on real GPUs. The agent develops hypotheses and submits recipes, while system services run training, protect evaluation, and link recipes, code, artifacts, and research notes into a persistent research record. This lets the agent work on the research process itself, from hypothesis formation and experimental design to result analysis and iterative refinement. Across six successive campaigns on eight-GPU nodes, Opus 5 improves Qwen3-4B-Base from to across seven benchmarks using a fixed, decontaminated data pool, against 70.4 for the official instruction-tuned model. We analyze the record of these campaigns: 839 accepted experiments, 711 evaluations, and nearly a thousand research notes. We observe that the agent keeps to a narrow band of recipes, changing one setting at a time over a few trusted corpora. Its largest losses rarely come from failed jobs, but from data it did not inspect and the intrinsic difficulty of post-training: the same recipe can help one checkpoint but hurt another. On the strategy side, it drops early conclusions that later evidence contradicts, while directions it rules out mostly stay ruled out.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent-Native Systems, LLMs, AutoResearch, Post-Training, Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ir96OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Athanor is an asynchronous infrastructure and research-record system for autonomous language-model post-training. A frontier agent proposes recipes involving SFT, RLVR, distillation, and merging, while a separate control plane executes jobs, protects evaluation, and preserves checkpoint and research lineage. The paper studies six successive campaigns with Claude Opus 5 improving Qwen3-4B-Base over a fixed pool of 264 corpora. Across 839 accepted experiments and 711 online evaluations, the best observed Core7 score increases from 26.36 to 63.21. The selected checkpoint scores 62.84 on final Core7 re-evaluation and 32.04 on the sealed OOD7 suite, compared with 15.34 for the base model on OOD7. The paper analyzes search locality, recipe convergence, data inspection, lineage dependence, and revision of research conclusions.

Critical strengths
  1. A substantial and practically relevant systems demonstration. The system connects executable recipes, asynchronous jobs, measurements, checkpoints, and research notes over hundreds of experiments (§§3–4). Failure recovery and persistent state address real obstacles to long-running autonomous research. The large improvement on Core7 is accompanied by improvement on a sealed suite, so the demonstrated value is not limited to the search feedback itself. The final re-evaluation and explicit accounting of inherited work make the evidence more interpretable.

  2. The research record enables unusually detailed analysis of unsuccessful search. The paper reports failed executions, non-improving checkpoints, repeated measurements, and revisions of the agent's hypotheses rather than only the winning recipe. Appendix E.4, for example, shows that one checkpoint's best observed score exceeds its repeat mean. Such records could become a useful community resource for studying search decisions and measurement noise, independently of whether the resulting model beats an official instruction-tuned model.

Critical weaknesses
  1. The experiments do not establish the incremental value of the proposed research infrastructure or agent strategy. The main references are the base model and official instruct model. There is no matched-budget simple recipe search or reduced-system agent comparison using the same data, defaults, and skills. All winning campaigns use a similar SFT-then-RLVR structure; meanwhile, Appendix D.2 states that authors supply backend defaults and eight research skills. The successive campaigns also change inherited information and checkpoints. The results therefore demonstrate that this configured agent can conduct useful post-training, but do not isolate how much the persistent record, control-plane separation, or agent-driven research contributes beyond a simpler search procedure. A bounded control using the existing campaign setting would substantially clarify the system's scientific contribution.

  2. Several central behavioral conclusions are stronger than their quantitative support. The assertion that unexamined data is the main source of costly failure relies on coverage counts and selected traces, without an aggregate comparison of losses or compute spent on inspected versus uninspected data. The key lineage example is also based on single evaluations: Table 14 reports +1.77 on one parent and −0.52 on another, while the paper uses 1.3 points as its operational noise threshold and documents substantial repeat variation. Lineage dependence is plausible, but the negative effect in that example is not clearly separated from evaluation noise. These issues matter because diagnosing the research bottleneck is a main contribution, not merely an illustrative anecdote.

Questions for authors
  1. Can a simple SFT/RLVR recipe sweep, or the same agent with a reduced research record, be compared under a matched budget and the same data/defaults/skills? What result would distinguish the benefit of Athanor's design from the strength of its starting research priors?
  2. How is “inspection” operationally identified, and what are the aggregate rates and magnitudes of degradation, with compute costs, for inspected and uninspected corpora? Can the Table 14 effects be accompanied by repeated or paired evaluations? If unavailable, would the authors frame these as hypotheses suggested by traces rather than established dominant causes?
  3. Can the full research skills, defaults, human operational interventions, data-filtering procedure, and executable records be made inspectable? Appendix D.3 states that overlapping benchmark questions were filtered but does not specify the matching rule or removed counts. These details determine both reproducibility and the extent of autonomous discovery.
4 comments
AuthorsOpenReview ↗

Reply to Reviewer ir96

Thank you for describing the research record as a potential community resource for studying search decisions and measurement noise.

Q1. A matched-budget simple recipe or a reduced system; what distinguishes Athanor's design from the strength of the starting priors.

  • Fixed recipe (general response, point 1): a strong hand-built SFT→RLVR→merge pipeline outscores Opus 5's campaigns, Opus 5.5 matches it in one day, and on a new target the agent's search far exceeds it (point 2).
  • Priors: the gains are the agent's own. Guidance ablation: guidance placed in the system prompt and guidance never opened give the same score (point 3 and Appendix A.1). On the upgraded suites, a team given no skills and no research advice reaches the instruct release (point 2). Data-pool ablation: given an empty pool, Opus 5.5 named eight public datasets from its own knowledge and scored 35.35 held-out, close to the 36.54–41.45 of runs with the pool (Appendix A.2).
  • System. The service ablation (reply to Reviewer g3y7, Q3) is a bounded control of the kind you describe, run on the upgraded suites: the same agent with the research services removed. Together these separate the two: the score does not depend on the priors we supplied, and Athanor is what lets the agent's research run and be checked.

The campaigns change inherited information and checkpoints by design, each to test one question, and each runs on one version of the environment's code; the pool, the decontamination index, the evaluator, and the goal stay fixed (§4.1; reply to Reviewer JQuP).

Q2. "Inspection", aggregate degradation, and Table 14.

  • §4.3 states all five findings as observations from the record, not controlled comparisons. Finding (3) states the observation itself: the costly losses are runs that completed correctly and made the model worse, not the 92 of 839 experiments that failed to run, and 12 of the 51 trained corpora were never inspected. A corpus counts as inspected when the researcher called the catalog's inspect tool on it; rows read through code are not counted, so 12 is an upper bound on the uninspected corpora. §4.3 notes that aggregate degradation and compute were not compared between inspected and uninspected corpora.
  • Table 14 of the submission (Table 16 in the revision): Table 15 gives single-reading noise, pooled over 30 checkpoints read more than once (83 readings): one reading has a standard deviation of 0.42, so the difference of two readings has about 0.59. These checkpoints were not re-evaluated; read against this noise, Parent A's +1.77 is about three standard deviations, and Parent B's −0.52 is within noise. Finding (4) reads: the same RLVR recipe raised one checkpoint and left a nearly identical one within noise; the abstract, §1, and Appendix B say the same, and the caption marks every entry as a single reading.

Q3. Skills, defaults, interventions, data filtering, records, and the decontamination rule.

  • Skills and defaults: Table 9 describes each of the eight skills, when the researcher loads it and what it covers; the guidance ablation shows the score does not depend on them. Appendix E.2 now states the backend defaults, which are the same in every campaign.
  • Decontamination (Appendix E.3): training rows and benchmark items are normalized (lowercase; punctuation and LaTeX removed; whitespace collapsed). A row is removed when its word-level 13-grams shared with an item of any of the 14 Core7 and OOD7 benchmarks cover at least half of that item's 13-grams; coverage is on the item's side, so an item embedded in a long row is caught. The check runs each time a data mix is built. Across the six campaigns, 292 builds checked 36.0M rows and removed 5,415 (0.015%) from 25 corpora; 78% of them overlapped the held-out OOD7 suite.
  • Human operations: Appendix E.6 (reply to Reviewer g3y7, Q5).
  • Code and records: we plan to release the code and the campaign records; still in progress.

Dear Reviewer ir96,

As the discussion period closes on November 18, we wanted to check whether our response has addressed your questions.

We would be happy to clarify any remaining concerns and would appreciate it if you would consider updating your score in light of our response!

Thanks for the clarification. I would like to maintain my score.

Thanks for your appreciation! Wishing you a great day!

Reviewer_JQuPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose Athanor, a fully automated post training agentic system for improving performance on multiple downstream tasks at once given a base model over long multi-node raining sessions. A set of 7 commonly used eval tasks are used to explicitly optimize for and another set of 7 are used as an OOD benchmark. Findings and failure modes based on analysis from agentic traces are presented, insufficient exploration seems to be a theme.

Critical strengths
  • This work tackles a very important, practical challenge at scale. I appreciate the amount of computational resources used and the findings presented openly, especially the trajectory analyses.
Critical weaknesses
  • The clarity of writing is the largest weakness of this paper and really holds it back. There are many interesting findings from the Trajectory Analysis in section 4.3, however the presentation and organization detract from the insights provided. Likewise, I found the abstract and introduction hard to follow due to the writing, it was not clear what exactly the takeaways were.
  • Comparison with any other autoresearch systems is lacking, hence it is not clear if the findings/failure modes from the trajectory analysis section are unique to this harness/could be solved with a better harness. While I am sympathetic to the fact that presented experiments require a large amount of computation resources, even ablations of components of the proposed framework on smaller scale experiments would be informative.
Questions for authors

What delineates campaigns/why start a new campaign? Is this due to context overload/rot?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer JQuP

Thank you for valuing the scale of the experiments and the openly presented trajectory analyses.

Writing.

We revised the parts where the takeaways were hardest to find. §4.3 now gives each finding a heading that states it, followed by the numbers that support it (pp. 7–9); the abstract and §1 state the same five findings. §4.2 states the main result against a fixed pipeline in one table (Table 2), and §4.1 states what each campaign tests.

Comparison with other autoresearch systems; ablations.

We have no head-to-head comparison because no existing system targets this setting. Most prior systems post-train one specialist model or search one part of the pipeline (Appendix C, Table 5); Athanor's setting post-trains a raw base model into one generalist checkpoint, scored on 14 benchmarks including a sealed held-out suite, over days on multiple nodes. Running another system in this setting would amount to re-implementing it, and the comparison would test our port rather than the system. Following your suggestion, we ran smaller-scale ablations of the framework's components (Appendix A), alongside a strong fixed-pipeline baseline (general response, point 1):

  • Guidance ablation: runs of the same agent that differ only in research guidance give the same score, so the gains are the agent's own (point 3).
  • Service ablation: with only a shell, the GPUs, and the evaluator, 3 of 8 runs never reach RL, against none of 9 with the services (reply to Reviewer g3y7, Q3).
  • Data-pool ablation: an empty pool in place of the curated one (reply to Reviewer ir96, Q1).

On whether the failure modes are specific to this harness: the narrow search follows the model, not the interface (Appendix A.1). In a leaner interface without the typed submit interface and the charter, three 24-hour Opus 5 runs reach 60.27–61.26; one reached 60.27 at hour 10 and stayed between 59 and 60 afterwards, close to where C1 and C2 ended. In the same interface, Opus 5.5 reaches 65.2–65.6. Changing the interface does not move the plateau; a stronger agent does.

Q. What delineates campaigns? Is it context overload?

It is not context. Every researcher session starts as a fresh process with no context carried over, and the record is its only continuity (Appendices D.3 and E.2). Campaigns are delineated by starting conditions, each testing one question: C1 and C2 start from the base without seeing each other, to test whether independent searches converge (finding 2); C3 inherits the records of C1–C2 but no checkpoints, to test what the record carries; C4–C6 also inherit retained checkpoints, to test continued search. Each campaign also runs on a single version of the environment's code, so that its ledger comes from one version. C5 and C6 ended by the plateau rule (20, later 30, measured checkpoints without improvement). The data pool, the decontamination index, the evaluator, and the goal stay fixed throughout. The revised §4.1 states this.

Dear Reviewer JQuP,

As the discussion period closes on November 18, we wanted to check whether our response has addressed your questions.

We would be happy to clarify any remaining concerns and would appreciate it if you would consider updating your score in light of our response!

Thank you for the additional experiments and response.

I appreciate the comparison with a fixed pipeline in Table 2 and the additional ablation experiments in Appendix A. However, it seems that the presentation of results of the ablation experiments consists entirely of a few sentences in Appendix sections A.1 and A.2, without a dedicated figure or table systematically summarizing the ablation conditions and outcomes. I found the interpretations of these results difficult to follow and several of the comparisons are summarized primarily through aggregate benchmark scores rather than per task. Providing a clearer breakdown of the results for each ablation condition would substantially improve their interpretability.

Additionally, while I appreciate the edits to the abstract and introduction and the additional headings in section 4.3, my main concerns on the clarity of presentation remain largely unaddressed. The paper introduces many system components and results, however understanding the motivation and interpreting each of these remains challenging. I therefore believe the paper would benefit from another round of more structural revisions to the current presentation rather than more lightweight edits.

Given the current state of the manuscript, I cannot recommend acceptance and therefore maintain my score. That being said, I want to emphasize that I appreciate the scale of the empirical study and would find the resulting research record very valuable. I would thus like to see the results communicated in a form where the contributions are easy to interpret and evaluate.

AuthorsOpenReview ↗

Reply to Reviewer JQuP: a table of the ablation conditions, and per-task results

Thank you for the specific direction. We have made both changes in the revised PDF.

1. One table of the ablation conditions and outcomes (new Table 4, Appendix A).

ArmAgentRunsReached RLheldout6Range
Research services present or removed
With the servicesOpus 4.83333.1726.02–38.48
Shell, GPUs, evaluatorOpus 4.83224.0520.75–27.56
With the servicesOpus 52235.0032.77–37.23
Shell, GPUs, evaluatorOpus 51033.71–
With the servicesOpus 5.53338.2236.54–41.45
Shell, GPUs, evaluatorOpus 5.53236.6230.79–40.44
One part of the setting removed, services kept
Every reference hiddenOpus 4.81137.89–
Empty data poolOpus 5.51135.35–

2. Per-task results (new Tables 5 and 6). Table 5 gives each of the 17 ablation runs above its score on all six heldout6 tasks. Table 6 does the same on the paper's held-out suite, OOD7, for the two guidance arms, a third Opus 5.5 run, the fixed pipeline of Table 2, the base, and the instruct release. Reading them changed what we say:

  • Removing the services costs Opus 4.8 ground on four of the six tasks — ALFWorld −25.5, ZebraLogic −15.7, IFBench −10.4, math −4.6 — and leaves LiveCodeBench and SuperGPQA within 1.5 points.
  • For Opus 5.5 the two arms are close in total (1.60) but not task by task: ALFWorld differs by 15.0 in the reduced arm's favour, LiveCodeBench by 3.9 the other way. We now state this instead of the total alone.
  • On OOD7 the agents' recipes and the hand-built pipeline tie in total (within 1.0) and differ task by task, by up to +7.0 on MATH-500 and −13.7 on LiveCodeBench. The two guidance arms differ by 0.86 in total and by at most 5.3 on any task.
  • Given an empty data pool the agent lands above the curated-pool runs on SuperGPQA and below them on IFBench.

3. On the presentation. Thank you for your valuable feedback. What we read you as asking for is that §3 introduce each service by the failure it prevents instead of by its name, and that the record and its five findings come ahead of the system description, so the paper opens with what was measured and brings in a component only where a finding needs one. We will carry it into the camera-ready, and would welcome your view on that order.

The three new tables shift the later appendix tables by three: the skills table is now Table 12, the human-operations table Table 15, the evaluation-noise table Table 18, and the lineage table Table 19.

Reviewer_g3y7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Athanor supports autonomous post-training research by separating an AI researcher’s decisions from asynchronous experiment execution and protected evaluation. It maintains a persistent record connecting recipes, checkpoints, measurements, and research notes. The authors demonstrate model improvements through multiple GPU campaigns and analyze the agent’s research behavior.

Critical strengths

Relevant operational problem. Sustaining research across delayed experiments, failures, and fresh agent sessions is a meaningful challenge. The working implementation demonstrates a consequential application beyond a short agent demonstration.

Inspectable research history. Connecting experimental artifacts with contemporaneous interpretations provides useful evidence about failures, revisions, and search behavior that final-recipe reporting would miss.

Critical weaknesses

The technical novelty is insufficiently established. Immutable submissions, persistent records, dependency scheduling, asynchronous execution, and recovery are familiar systems mechanisms. The paper does not clearly identify which limitation requires a new mechanism or demonstrate a distinctive benefit from their integration. Feature breadth alone is insufficient to establish the systems contribution.

The added value is not isolated. Improvement over a raw base model establishes that the overall process produces useful post-trained models. It does not establish how much adaptive search contributes beyond a competent training recipe, or how much Athanor contributes beyond conventional infrastructure. The observational findings are useful, but currently provide limited evidence of a broadly better research process.

Questions for authors

What is the central technical contribution beyond integrating established workflow, provenance, scheduling, and evaluation mechanisms? Which concrete limitation does it overcome?

How does the autonomous search compare with a strong fixed SFT→RLVR pipeline or a simple recipe sweep under the same data access and compute budget?

What measured advantage does Athanor provide over the same researcher using a conventional scheduler and experiment tracker—for example, in research progress, resource efficiency, or recovery?

What evidence isolates the benefit of linking research notes to checkpoint lineage, compared with retaining ordinary experiment results and notes? A focused comparison would help substantiate this distinctive design choice.

How much human operational work was required during the campaigns, including interventions, recovery, and maintenance? Please distinguish this from initial environment preparation so the practical degree of autonomy can be assessed.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer g3y7

Thank you for recognizing that sustaining research across delayed experiments, failures, and fresh sessions is a meaningful challenge, and that the inspectable history captures what final-recipe reporting would miss.

Q1. The central technical contribution.

We do not claim the individual mechanisms are new. The challenge Athanor addresses is making it practical for an agent to run generalist post-training research for days on multiple nodes, and for a reader to trust and audit what it found. Athanor provides both at once. Convenience for the agent: an experiment is one command; the environment schedules, recovers, and evaluates it, so research ran for days unattended (§4.4: all but one of 1,381 launches ended in a recorded terminal state; the failures of 124 of 402 sessions stayed local). Auditability for the reader: an evaluator the agent cannot modify, a held-out suite it never sees, and every result tied to its recipe, data, and parent checkpoint. What this enables: 839 experiments over six campaigns, and five findings read directly from the record (§4.3). Appendix C now compares Athanor with neighboring systems by the setting each measures, not by feature coverage.

Q2. General response, points 1–2: on the paper's suites Opus 5.5 matches a strong fixed SFT→RLVR pipeline in one day, and on a new target the agent's search reaches 66.1 / 46.7 where the same pipeline reaches 41.2 / 24.2. We did not run a recipe sweep; the fixed pipeline and the agent runs bracket the compute budget rather than match it.

Q3. Advantage over a conventional scheduler and experiment tracker.

We ran this as an ablation (Appendix A.2): the same agent with the environment's services, or with only a shell, the GPUs, and the frozen evaluator. Held-out evaluation is identical in both. With the services, all 9 runs reached RL; without them, 3 of 8 never did. For Opus 4.8 the services also raise the mean held-out score by 9.12 points; for Opus 5.5 the difference, 1.60, is within its run-to-run spread. The measured advantage is that research progresses reliably, and that a weaker agent scores substantially higher. The reduced arm has less than a scheduler and a tracker and keeps the sealed evaluation, so the ablation isolates the service layer.

Q4. Linking research notes to checkpoint lineage.

The linkage serves two purposes. Audit: every claim in a note is tied to the checkpoint and measurement behind it, so a reader can check what the agent believed against what happened; every finding in §4.3 is read from these linked records. Externalized memory: each session starts fresh with no context carried over, so the linked record is what lets a new session or campaign recover the state of the research and resume where the last one stopped. We have no controlled comparison of its effect on scores; it made the following possible. C3 inherited only the records of C1–C2 and reached 61.67 in 15.1 hours (1,187 GPU-h), against 59.51 in 23.3 hours (915 GPU-h) for C1 from scratch (one campaign each, on different environment versions). C6 reopened a closed direction because the record kept the condition under which it had been closed (failure on RL-trained checkpoints only), and the reopened stage gained 4.0 points on a fresh SFT checkpoint.

Q5. Human operational work (Appendix E.6, Table 12).

  • Before launch: provisioning the nodes, deploying the environment, opening each campaign, and, for C4–C6, seeding the retained checkpoints.
  • During the runs: no human submitted, withdrew, or annotated an experiment, and the ledgers record no human message to the researcher; apart from the run-state commands below, all 2,583 commands in the six ledgers come from the researcher. Maintenance consisted of 16 configuration redeployments while a campaign ran (C1 9, C4 3, C5 1, C6 3; in C1, where the ledger records the deployed parts, the charter and skills were unchanged each time) and 10 run-state commands: ending C1, C3, and C4 at version boundaries, five for C2 (ended three times, resumed twice), and pausing C6 and resuming it with the plateau threshold raised from 20 to 30.
  • C5 and C6 were ended by the system's plateau rule.

Dear Reviewer g3y7,

As the discussion period closes on November 18, we wanted to check whether our response has addressed your questions.

We would be happy to clarify any remaining concerns and would appreciate it if you would consider updating your score in light of our response!

Thanks for the detailed rebuttal. The comments answered some of my questions, e.g., the benefit of the system through more baseline comparisons. But my main concern on research contribution and what the system adds beyond a competent assembly of existing infrastructure remains. I therefore maintain my current rating.

AuthorsOpenReview ↗

Reply to Reviewer g3y7: the size of what the assembly adds

Thank you for reading the response and for saying which part of it landed.

On the remaining concern we would rather be concrete about what we claim. We do not claim that the individual mechanisms are new. We claim a setting that was not previously measured, and a measured effect of assembling these mechanisms for it: a generalist agent post-training a raw base model for days on real GPUs, scored by an evaluator it cannot modify on a suite it never sees.

The measured part is now one table in the revised PDF (Table 4, Appendix A). With the research services, every one of 9 runs reached RL. With only a shell, the node's GPUs and the same frozen evaluator, 3 of 8 did not, and the mean held-out score of Claude Opus 4.8 falls by 9.12 points, from 33.17 to 24.05. For Opus 5 the difference is 1.29 and for Opus 5.5 it is 1.60, inside the run-to-run spread: the assembly matters most for the weaker agent and fades as the agent improves, which is itself a result about when this infrastructure is needed. Table 5 now gives every one of those runs task by task, and Table 6 does the same on the paper's held-out suite.

Two further arms separate the assembly from what we supplied with it. An agent that kept the services but was shown no reference measurement scored 37.89, at the top of the 26.02-38.48 of the runs with references; an agent given an empty data pool named eight public datasets from its own knowledge and scored 35.35, against 36.54-41.45 with our curated pool. So the score is not carried by the information we provided; what the environment buys is that long runs complete, recover, and can be checked afterwards.

That is the contribution we are claiming. We are glad to answer anything further before the discussion closes.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2): what Athanor contributes, and where agent search adds value

We thank the reviewers. Reviewer g3y7 finds the operational problem relevant and the inspectable research history useful; Reviewer JQuP values the computational scale and the openly presented trajectory analyses; Reviewer ir96 calls the work a substantial and practically relevant systems demonstration whose research record "could become a useful community resource".

What the paper contributes.

Athanor makes long-horizon post-training research practical for an agent and auditable for a reader.

Practical: the agent submits an experiment as one command and the environment runs, recovers, and evaluates it, so research continues for days without a human (§4.4: all but one of 1,381 launches ended in a recorded terminal state, and failures stayed local).

Auditable: every score is tied to its recipe, parent checkpoint, data, and decontamination, measured by an evaluator the agent cannot modify, with a held-out suite it never sees.

Findings at scale: in it, Opus 5 ran six campaigns and 839 experiments post-training a raw base model, and the record yields five findings about how the agent does research (§4.3). We have kept developing Athanor and running newer agents in it; the results below answer how much the agent's search adds beyond a competent recipe.

1. On the paper's suites, the agent reaches a strong hand-built pipeline (§4.2, Table 2).

Same base, same frozen Core7/OOD7 evaluator; the pipeline trains on public corpora from the same sources, decontaminated with the same rule.

MethodGPU-hCore7OOD7
Qwen3-4B-Base–26.3615.34
Hand-built pipeline: SFT soup → GRPO on math, code, and mixed prompts → TIES merge≈930 training, plus human design65.1638.47
Opus 5, six campaigns (best checkpoint; this paper)5,814 job time63.2132.04
Opus 5, one 24-h run on one 8-GPU node (3 runs; OOD7 for 2)192 each60.27–61.2631.60–32.54
Opus 5.5, one 24-h run on one 8-GPU node (3 runs)192 each65.18–65.6038.31–39.46
Qwen3-4B-Instruct-2507–70.3645.27

Opus 5.5 matches the pipeline in a single day on one node, with about a fifth of its GPU hours and no human recipe design; Opus 5's campaigns stay below it.

2. On a new target, the agent's search far exceeds the fixed recipe and reaches the instruct release (Appendix A.2).

On upgraded evaluation suites (general response 2/2), the same pipeline scores 41.2 dev6 / 24.2 heldout6, while a team of four Opus 5.5 agents scores 66.1 / 46.7 under the same protocol, the one the agents worked with. Under the final evaluation protocol the team raises Qwen3-4B-Base from 9.2 to 48.2 on heldout6, against 47.9 for Qwen3-4B-Instruct-2507, and on three more bases a single Opus 5.5 run exceeds the vendor's instruct release on heldout6. A fixed recipe stays where it was built; the agent re-derives its recipe for each target. This is the adaptive-search value Reviewer g3y7 asks about.

3. The gains are the agent's own (Appendix A.1–A.2).

Guidance placed in the prompt and guidance never opened give the same score, and an empty data pool scores close to runs with our curated pool (35.35 vs 36.54–41.45), so the result is not inherited from the priors we supplied (reply to Reviewer ir96).

AuthorsOpenReview ↗

General response (2/2): the upgraded evaluation and the runs added after submission

Upgraded evaluation.

As the agents improved, Core7 could no longer separate the strongest runs: three Opus 5.5 runs lie within 0.42 points of each other, inside the single-reading noise measured in Table 15. We therefore upgraded the evaluation for the new campaigns: dev6 for development (five 2025 math competitions including AIME and HMMT February, one half of ZebraLogic, LiveCodeBench 2024, MMLU-Pro, IFEval, four ALFWorld task types) and heldout6 held out (AIME 2026, HMMT February 2026, HMMT November 2025, the other half of ZebraLogic, LiveCodeBench 2025, SuperGPQA, IFBench, two unseen ALFWorld task types). Every sealed score is avg@4, the mean over four evaluation seeds.

The environment was also streamlined: the agent drives the training starters and services directly, while the frozen evaluator, the decontamination rule, and the record are unchanged. Opus 5 reaches the same plateau in it, so the plateau follows the agent, not the interface (reply to Reviewer JQuP). Numbers on the upgraded suites are never compared with Core7/OOD7. All of this is in the new Appendix A.

What the added runs answer.

Question (reviewer)RunResultWhere
Does agent search match a strong fixed recipe? (g3y7 Q2, ir96 Q1)Fixed pipeline, Opus 5, Opus 5.5 on Core7/OOD7Opus 5.5 matches it in one day, with a fifth of its GPU hours§4.2, Table 2
Does adaptive search add value beyond a recipe? (g3y7)The same pipeline and an Opus 5.5 team on the upgraded suites41.2 / 24.2 against 66.1 / 46.7App. A.2
How far can the agent go? (ir96 Q1)Team of four Opus 5.5 agents, 104 h, no skills or research advice9.2 → 48.2 heldout6, at the instruct release (47.9)App. A.2
Does it generalize across bases?One 72-h Opus 5.5 run each on three more basesAbove the vendor's instruct release on all threeApp. A.2
Ablation: are the gains inherited from our guidance? (ir96 Q1)Two Opus 5.5 runs differing only in guidance; Opus 5 without skillsSame score either wayApp. A.1
Ablation: is the curated pool a hidden prior? (ir96 Q1)Opus 5.5 with an empty pool35.35, close to 36.54–41.45 with the poolApp. A.2
Ablation: what do the services add? (g3y7 Q3, JQuP)Same agent with only a shell, the GPUs, and the evaluatorAll 9 runs with services reach RL, 3 of 8 without do not; +9.1 held-out for Opus 4.8App. A.2
Are the failure modes specific to this harness? (JQuP)Opus 5 in the streamlined environmentSame plateau, 60.3–61.3App. A.1

Where the revision answers each point.

Added text is blue; text removed at a reviewer's request is struck through in red.

  • §4.1: what each campaign tests (JQuP, ir96). §4.2: Table 2 (g3y7, ir96).
  • §4.3: each finding under a heading that states it, all five stated as observations from the record (JQuP, ir96).
  • Table 16 (Table 14 in the submission): every entry marked as a single reading and read against the noise of Table 15 (ir96).
  • Appendix A (new): the runs above. Appendix C: neighboring systems compared by the setting each measures (g3y7).
  • Appendix E.3: the decontamination rule and counts; E.6: human operations; E.2: the backend defaults (ir96, g3y7).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.