- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work presents Singularity-Bench, a benchmark that consists of 5 tasks in the training and production of an agentic LLM. The authors evaluated 12 model-harness pairs on the benchmark and found that Claude Opus 5.5 with Claude Code CLI is competitive with the provided human baseline. The authors claim that the frontier models may have already reached "singularity", able to take end-to-end responsibility for building stronger successors models.
Critical strengths
- The authors construct five carefully curated tasks that encompass real-world complexity, a strong human baseline, and reliable scoring, with a strong system engineering focus.
- The authors present decent analysis on why Cloud Opus 5.5 performs better than other tested agents in the task, giving insight for direction to improve in other agentic models.
Critical weaknesses
- "Singularity" typically require an ultra-intelligent machine "design even better machines" (I. J. Good, 1965). But in terms of algorithm and model architecture, the benchmark values exploitation of domain knowledge rather than exploring novel methods.
- In the pre-training task, building the system and conducting the training runs share the same time frame. The AI agent has to balance the wall time of building a better framework, finding a better recipe and the main training. High performing agents start only one main training run very early on, leaving little time for architectural and parameter decisions. Proposing and testing novel model architectures or training algorithm is therefore too risky to even attempt.
- In the post-training task, the algorithm is restricted to RL with SFT banned. Other post-training methods, such as self-distillation or other novel new paradigm methods the agent might propose, are unspecified.
- All evaluations were conducted only once, with no variance analysis. Although the main conclusion, the high performance of Opus 5.5 compared to other models is unlikely to be because of pure luck.
Questions for authors
-
Please address my first weakness comment, or reframe the contribution of the benchmark as mainly a coding and engineering focused test on autonomous AI-system development.
-
In the
coding-agent-harnesstask, what if the model just copies and submits the harness where the models themselves were running in (e.g. claude code or codex cli)? The hack provides at least baseline performance. What if the model hard codes the recommended solution in the harness prompt since there is no hidden test case? -
Please disclose the reasoning effort of the models being tested.
3 comments
2. Repeated runs (W2).
now reports three end-to-end pretraining runs for each of four open-weight models using Codex: Kimi K3, GLM 5.3, DeepSeek V4.1 Flash, and MiniMax M3. Run 1 is the result in ; Runs 2 and 3 are fresh agent runs from the starting codebase and data, repeating design, implementation, training, and submission under the same twelve-hour, four-GB300 budget and provider-default reasoning effort (high).
Each cell below reports PPL / cross-entropy (authoring API cost). Cross-entropy is ln(PPL), in nats per token. Score means exclude failed runs; mean API cost includes all three runs.
| Model + harness | Run 1 | Run 2 | Run 3 | Mean |
|---|---|---|---|---|
| Kimi + Codex | 14.431 / 2.6694 ($12.40) | 14.809 / 2.6953 ($14.49) | 17.838 / 2.8813 ($17.28) | 15.69 / 2.7487 ($14.72) |
| GLM + Codex | 16.311 / 2.7918 ($8.59) | 16.966 / 2.8312 ($7.14) | 16.603 / 2.8096 ($7.41) | 16.63 / 2.8109 ($7.71) |
| DeepSeek + Codex | 17.582 / 2.8669 ($0.54) | 16.015 / 2.7735 ($0.49) | 18.786 / 2.9331 ($0.50) | 17.46 / 2.8578 ($0.51) |
| MiniMax + Codex | Fail / — ($0.44) | 21.514 / 3.0687 ($3.69) | 18.257 / 2.9046 ($2.07) | 19.89 / 2.9867 ($2.07) |
Both mean PPL and mean cross-entropy preserve the first-run ordering: Kimi, GLM, DeepSeek, then MiniMax, with failed submissions ranked last. MiniMax's first run ends without a checkpoint; its two successful repetitions still have the highest mean PPL and cross-entropy in this cohort.
The cross-entropy sample standard deviations are 0.1156, 0.0197, 0.0802, and 0.1160 nats/token, respectively, around mean losses of 2.7487, 2.8109, 2.8578, and 2.9867. The corresponding PPL standard deviations are 1.87, 0.33, 1.39, and 2.30. MiniMax's score means and standard deviations use its two successful runs. Reporting the individual runs and both scales quantifies variability while showing that the average performance ordering is consistent with the first-run comparison for these four configurations.
We plan further repetitions across the remaining agents and tasks. Limited compute availability currently prevents completing this broader evaluation.
3. Copying the installed harness or hard-coding answers (Q2).
The reviewer raises a good point, and we accounted for this by implementing a safeguard that prevents the agent from accessing the installed authoring harness. Command and file tools run under a separate worker identity whose filesystem permissions prevent reading or executing the private CLI installation. Installed coding-agent harnesses are also unavailable during downstream evaluation. The reported runs were therefore prevented from copying or invoking the installed harness to obtain baseline performance. We have made this explicit in and .
Downstream task identities and one of the three driver models are withheld during harness construction. The downstream verifiers include hidden correctness tests, and source and trajectory review checks for task-specific answer injection and unauthorized model access. General prompting strategies, such as requiring tests before completion, are legitimate harness design; embedding a particular evaluation task's solution is not.
4. Reasoning effort (Q3).
, , and now report authoring settings across all six tasks. Open-weight models use the provider default, confirmed to be high. Opus 5.5, Opus 5, and GPT-6.1 Sol use their strongest setting, max. These settings describe the authoring agents; the safeguard-validation judges use medium effort.
Official Comment by Reviewer jK2w
Thank you for the detailed response and the revision. The reframing in the abstract, introduction, Figure 1 and conclusion addresses my first concern: described as a benchmark of autonomous AI-system development under fixed budgets, the tasks measure what they measure, and I find that contribution worthwhile. The reasoning-effort disclosure (Appendix A.3, Table 7) and the repeated pre-training runs (Appendix F) are appreciated.
On the harness task, the worker-identity isolation, hidden correctness tests and withheld task identities described in Section 3.1 and Appendix J.1 resolve the specific shortcuts I raised. I note that none of these protections appeared in the submitted version, so the red-team exercise reported in Section 3.2 did not cover them.
I also note for the record that the revised pre-training task runs on four GPUs rather than two, so every pre-training score and the human baseline in Table 4 differ from the submitted version, and the post-training task has been replaced. These are improvements, but a substantial part of the reported results is new in this revision. I keep my score.
Thank you for recognizing the realistic tasks, engineering depth, and solution analysis. The revision evaluates eleven model–harness configurations on all six tasks, incorporates data curation and the new terminal-agent post-training results, and highlights changes in blue.
1. Scope, algorithmic novelty, and the development budget (W1/Q1).
We have clarified the contribution as a benchmark of autonomous AI-system development: selecting, implementing, and integrating methods into working systems under realistic resource constraints. Architecture, model size, optimizer, precision, and parallelism remain open in pretraining. Sharing a deadline between development and training tests the consequential decision of how much experimentation is worth its cost.
The hardware-budget comparison in Section 4.2 (p. 9) and Appendix E.1 (pp. 20–21) makes this decision-making concrete. In separate twelve-hour runs on two and four GB300 GPUs, Opus 5.5 chooses 20 and 24 layers, respectively, at the same hidden width of 1,536. The resulting models have 0.566B and 0.680B non-embedding parameters, train on 12.43B and 21.58B tokens, and reach PPLs of 13.783 and 12.769. Both trajectories show throughput experiments before model selection: the two-GPU run compares widths 1,536 and 2,048; the four-GPU run tests three widths and measures multi-GPU scaling. Main training starts at 39 and 44 minutes, respectively, after these experiments. The early start therefore follows empirical design choices within the time budget.
The revised Introduction (pp. 1–2) and Figure 1 (p. 2) distinguish the motivating successor-development loop from the individual stages evaluated here. Repeated frontier-scale training and deployment cycles would require substantially greater resources; these tasks provide tractable measurements of the capabilities needed to undertake them.
Post-training now evaluates terminal-use agents: an agent builds the training system and improves a roughly 30B-parameter GLM-4.7-Flash model on four GB300s. Objectives, trainable parameter subsets, and rollout/training allocation are open choices. Self-distillation is permitted, including supervised training on the target model's own task-generated traces. None of the reported agents uses it; all choose policy-gradient RL. We make this scope explicit in Section 3.1 (pp. 3–6) and Appendix H.1 (p. 25).
The restriction concerns external supervision. The task provides training environments and rewards, but no supervised solution traces. Allowing the authoring agent to supply its own answers as SFT targets would transfer its task-solving ability into the trained model's score. We treat this shortcut as reward hacking under the task's contract, so demonstrations generated by the authoring agent or another external teacher are prohibited. Self-distillation uses the target model's own experience.
The new midtraining data-curation task, integrated into the main benchmark (Section 3.1 (pp. 3–6), Section 4.1 (pp. 8–9), Table 4 (p. 8); Appendix G (pp. 23–24)), covers SFT after pretraining using supplied corpora. Agents select, filter, format, and mix data from 1.1 TB of corpora and may use continued pretraining, SFT, RL, or combinations. Every checkpoint then undergoes the same downstream RL stage. Together, the tasks cover both learning from curated demonstrations and improving a policy through its own experience.