acceptodds
Withdrawn from ICLR 2027

Fathom-Autoresearch: Towards Reliable Autonomous Post-Training Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Autonomous post-training has a fixed deadline but delayed task-level feedback. An experiment can guide the remaining campaign only if its intervention is realized, exposed adequately, and evaluated in time to change the next decision. We pro- pose Fathom-Autoresearch, which couples evidence-grounded research reviews to continuous implementation and checkpoint retention. Across seven PostTrainBench tasks, its selected-checkpoint aggregate is 44.4% for Qwen3-1.7B-Base and 54.4% for Qwen3-4B-Base. It exceeds the persistent Codex CLI-Kimi K3 control in all eight displayed matched task model pairs. Across four tasks per model, effective GPU hours spent on selected-checkpoint work or completed evaluation are 16.7 versus 12.8 at 1.7B and 12.0 versus 8.7 at 4B. A frozen-state replay favors the Fathom-Autoresearch Planner’s written experiments by 2.12 points on a 24-point rubric across 15 states, without testing online execution. Together these layers characterize finite-horizon experimental control rather than checkpoint score mediation by any single role.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
LLM agents, automated AI research, LLM post-training, resource-aware planning, agentic research harness, agent harness, recursive super intelligence

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_NDSqOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Fathom, a two-role agent system for autonomous LLM post-training under a ten-hour budget: a fresh Planner reviews recorded evidence and publishes the next experiment, while a persistent Executor implements it, with a shared filesystem as the research record. On PostTrainBench-derived tasks with Qwen3 base models, it reports higher scores than a single-agent Codex control on four tasks and a higher rubric score for its Planner’s written plans in a frozen-state replay.

Critical strengths
  1. The paper is candid about the scope of its evidence, stating openly in the limitations and conclusion that the results do not show either role to be individually causal.
  2. The evaluation covers several complementary angles, including end-to-end scores on two model sizes, GPU-hour allocation, trajectory case studies and a frozen-state replay of planning decisions.
Critical weaknesses
  1. Limited novelty. The framework is a planner–executor split with a shared filesystem record, which the paper itself says is neither the first such split nor a new search algorithm (Sec. 2), so the contribution reads as prompt and harness engineering rather than a new method.
  2. No component ablation. No experiment removes or varies the fresh Planner, the persistent Executor or the filesystem record, and the limitations state that the process campaigns used revision-scoped Executors rather than the persistent-Executor design the paper describes.
  3. Unfair comparison with the baseline. Appendix C admits that some Codex–Fathom pairs in Table 2 were run under different access conditions (with or without the literature pack, with or without a teacher model), without saying which side had more, so the score gaps cannot be attributed to Fathom’s design.
Questions for authors
  1. What does Fathom add beyond the planner–executor designs of ARTS, Curie and R&D-Agent, and which results show that this difference matters?
  2. Please add ablations on the four matched tasks: a persistent Planner instead of a fresh one, the Planner removed, and the Fathom Planner instructions given to the single Codex agent.
  3. Table 2 should be re-run with every Codex–Fathom pair under the same literature and teacher access, with the access condition listed per cell.
  4. How many runs underlie each cell of Tables 1 and 2, and what are the empirical standard deviations rather than the presentation estimates?
Reviewer_ft2iOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes FATHOM, a multi-agent autoresearch system for autonomous LLM post-training. A planner proposes a research plan at every review and the executor implements it, with information communicated between the two via filesystem. The authors evaluate their autoresearch system on PostTrainBench, using Qwen3-1.7B/4B as the model to post-train and Kimi-K3 as the researcher model.

Critical strengths
  • The paper attacks an important problem, that autoresearch agents commit to an initial strategy and rarely revise it.
Critical weaknesses
  • The paper is very hard to follow. The writing is dense, unreadable and full of undefined jargon. eg: supplied mean (variance). Also, the notation in Section 3, around Equation 1, has not been defined. Also, this equation is never used anywhere else.
  • The authors present uncertainty estimates in Tables 1 and 2. But they state that these are “presentation estimates rather than empirical run variance”. Presenting non-empirical numbers as uncertainty estimates is a serious integrity concern. The number of runs behind the tables is not disclosed.
  • The papers headline result in Table 1 is not "matched" as also stated by the authors. This calls the credibility of their results and evaluation into question.
  • Table 2 gains are unsurprising. Fathom's planner receives guidance that the Codex control is not described as receiving. The comparison should be run against more competent baselines from PostTrainBench, to showcase that their proposed approach has merits.
  • Novelty is limited. The authors themselves concede that their method is “neither the first scientist-executor split nor a new search algorithm”.
Questions for authors

See Weaknesses

Reviewer_1iqXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Doing post training successfully requires robust strategy planning, reliable execution and adapting based on execution feedback. For autonomous LLM post-training tasks, the paper argues that the main bottleneck for current AI research agents is experimental control under delayed feedback rather than code execution. Existing agents frequently suffer from 2 challenges:

  1. getting stuck to a strategy i.e. spending most of their compute budget to an initial training strategy before obtaining a complete evaluation and 2) context pollution i.e. low-level debugging logs clutter high-level scientific reasoning.

Proposed method FATHOM is a multi-agent system built on Codex CLI that separates planning from execution across two roles communicating solely through a shared filesystem (PLAN.md and EXECUTION.md) rather than a shared chat history. At review boundaries, a fresh, stateless Planner inspects recorded evidence, literature, and remaining time to publish an immutable scientific program (PLAN.md), while a persistent, stateful Executor continuously implements the authorized experiment, handles operational repairs, evaluates checkpoints, preserves the best model, and triggers an early review if empirical results contradict the plan. FATHOM is evaluated using Kimi-K3 to post-train Qwen3-1.7B-Base and Qwen3-4B-Base across seven PostTrainBench-derived benchmarks. Proposed method outperforms a persistent single-agent Codex-CLI/Kimi-K3 baseline in all eight matched comparisons across four tasks.

Critical strengths

This paper tackles a well-motivated problem of how to separate discovery loops from execution and how to provide insights to avoid agents getting stuck into strategy lock-in. Pairing an episodic fresh Planner with a persistent Executor over a shared file system directly addresses context asymmetry.

End-to-end scores, GPU-hour attribution, executed trajectories, and a controlled mid-trajectory frozen-state planning replay together leads to comprehensive evaluation. The Section 5.3 trajectory case studies (AIME early measurement; GPQA plateau to RFT switch with rollback; plus negative cases) nicely demonstrate the mechanism.

Critical weaknesses
  • The headline comparison in Table 2 is not strictly matched. Appendix C.5 says Codex–FATHOM pairs differ in their access to the curated literature pack and local 27B teacher model. It is not clear how runs were selected for Table 2. Similarly, comparing local runs against official PostTrainBench leaderboard entries (e.g., Locus) isn’t fair, FATHOM needs to account for the curation cost for pre-assembled DeepResearch corpus in the 10-hour budget.

  • It is hard to isolate the proposed fresh-Planner / persistent-Executor architecture from the post-training domain knowledge embedded in the specialized prompt. Giving the single persistent Codex agent the same detailed scientific guidance without role splitting can help with this. Finally, in the frozen-state planning study, the same model (Kimi-K3) both generates and judges the research plans.

  • Some inconsistencies in the experimental results need to be double-checked and fixed: Some inconsistencies in experimental results need to be double-checked and fixed. E.g.

    • Table 2 captions say "supplied mean (variance)" while Section 6 says these are "presentation estimates rather than empirical run variance" after "row normalization".
    • The Abstract says "effective GPU hours spent on selected-checkpoint work or completed evaluation are 16.68 versus 12.78 at 1.7B and 12.00 versus 8.67 at 4B." Whereas, Section 5.2 (Lines 355–361) explain additional numbers leading to confusion.
    • Confusion regarding internet access: Section 5 (L273) and Appendix B.1 state internet access was disabled, whereas the operative prompts in Appendix 7 state "Internet research is enabled for this campaign."
Questions for authors
  1. To better isolate the architectural contribution from prompt engineering, would it be possible to run an ablation where the single-agent Codex control receives the exact same detailed scientific guidance prompt as the FATHOM Planner?
  2. Have you considered using a different frontier model (e.g., GPT-4o or Claude 3.5 Sonnet) as the judge in the frozen-state planning study to rule out LLM self-preference bias?
  3. Could you provide more details on the curation of the offline dataset collection? Specifically, how did you ensure that the pre-vetted offline corpus is free from test-set data contamination, given that reward hacking is a major issue in this domain?
  4. In Table 1, what is the exact task-weighting formula behind the Avg column? Also, for Table 2, how were the reported runs selected from the archive, and which specific Codex–FATHOM pairs had matched access to the literature pack and local teacher model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.