- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Fathom, a two-role agent system for autonomous LLM post-training under a ten-hour budget: a fresh Planner reviews recorded evidence and publishes the next experiment, while a persistent Executor implements it, with a shared filesystem as the research record. On PostTrainBench-derived tasks with Qwen3 base models, it reports higher scores than a single-agent Codex control on four tasks and a higher rubric score for its Planner’s written plans in a frozen-state replay.
Critical strengths
- The paper is candid about the scope of its evidence, stating openly in the limitations and conclusion that the results do not show either role to be individually causal.
- The evaluation covers several complementary angles, including end-to-end scores on two model sizes, GPU-hour allocation, trajectory case studies and a frozen-state replay of planning decisions.
Critical weaknesses
- Limited novelty. The framework is a planner–executor split with a shared filesystem record, which the paper itself says is neither the first such split nor a new search algorithm (Sec. 2), so the contribution reads as prompt and harness engineering rather than a new method.
- No component ablation. No experiment removes or varies the fresh Planner, the persistent Executor or the filesystem record, and the limitations state that the process campaigns used revision-scoped Executors rather than the persistent-Executor design the paper describes.
- Unfair comparison with the baseline. Appendix C admits that some Codex–Fathom pairs in Table 2 were run under different access conditions (with or without the literature pack, with or without a teacher model), without saying which side had more, so the score gaps cannot be attributed to Fathom’s design.
Questions for authors
- What does Fathom add beyond the planner–executor designs of ARTS, Curie and R&D-Agent, and which results show that this difference matters?
- Please add ablations on the four matched tasks: a persistent Planner instead of a fresh one, the Planner removed, and the Fathom Planner instructions given to the single Codex agent.
- Table 2 should be re-run with every Codex–Fathom pair under the same literature and teacher access, with the access condition listed per cell.
- How many runs underlie each cell of Tables 1 and 2, and what are the empirical standard deviations rather than the presentation estimates?