acceptodds
Under review as a conference paper at ICLR 2027

Singularity-Bench: Can Agents Build Agents?

Abstract

Frontier AI agents increasingly succeed on realistic, long-horizon tasks and are beginning to improve the systems through which they operate. Yet existing benchmarks often restrict agents to narrow action spaces, use simplified environments, or rely on verification methods that are insufficient for arbitrary agent-generated systems. We introduce SINGULARITY-BENCH, an open-ended benchmark for autonomous AI development with five end-to-end tasks spanning pre-training, post-training, inference, cluster orchestration, and the agent harness. Each task requires agents to implement core systems and evaluates them against mature human-engineered baselines on identical hardware. We also develop an evaluation framework that combines isolated execution, protected logging, and independent judges to detect invalid shortcuts. Across 12 agents built from 8 frontier models and 3 harnesses, Opus 5.5 with Claude Code is the only agent to outperform the human baseline on all five tasks, achieving a singularity index of 1.275, suggesting that it might already reach singularity. The remaining agents average only 0.56, with 8 of 12 scoring zero on inference, revealing highly uneven progress. Our vision is for SINGULARITY-BENCH to track when agents reach the singularity: the point at which they transition from assisting AI development to taking end-to-end responsibility for building stronger successors.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Benchmark, Recursive Self-Improvement, Agent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jK2wOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work presents Singularity-Bench, a benchmark that consists of 5 tasks in the training and production of an agentic LLM. The authors evaluated 12 model-harness pairs on the benchmark and found that Claude Opus 5.5 with Claude Code CLI is competitive with the provided human baseline. The authors claim that the frontier models may have already reached "singularity", able to take end-to-end responsibility for building stronger successors models.

Critical strengths
  1. The authors construct five carefully curated tasks that encompass real-world complexity, a strong human baseline, and reliable scoring, with a strong system engineering focus.
  2. The authors present decent analysis on why Cloud Opus 5.5 performs better than other tested agents in the task, giving insight for direction to improve in other agentic models.
Critical weaknesses
  1. "Singularity" typically require an ultra-intelligent machine "design even better machines" (I. J. Good, 1965). But in terms of algorithm and model architecture, the benchmark values exploitation of domain knowledge rather than exploring novel methods.
  • In the pre-training task, building the system and conducting the training runs share the same time frame. The AI agent has to balance the wall time of building a better framework, finding a better recipe and the main training. High performing agents start only one main training run very early on, leaving little time for architectural and parameter decisions. Proposing and testing novel model architectures or training algorithm is therefore too risky to even attempt.
  • In the post-training task, the algorithm is restricted to RL with SFT banned. Other post-training methods, such as self-distillation or other novel new paradigm methods the agent might propose, are unspecified.
  1. All evaluations were conducted only once, with no variance analysis. Although the main conclusion, the high performance of Opus 5.5 compared to other models is unlikely to be because of pure luck.
Questions for authors
  1. Please address my first weakness comment, or reframe the contribution of the benchmark as mainly a coding and engineering focused test on autonomous AI-system development.

  2. In the coding-agent-harness task, what if the model just copies and submits the harness where the models themselves were running in (e.g. claude code or codex cli)? The hack provides at least baseline performance. What if the model hard codes the recommended solution in the harness prompt since there is no hidden test case?

  3. Please disclose the reasoning effort of the models being tested.

3 comments

Thank you for recognizing the realistic tasks, engineering depth, and solution analysis. The revision evaluates eleven model–harness configurations on all six tasks, incorporates data curation and the new terminal-agent post-training results, and highlights changes in blue.

1. Scope, algorithmic novelty, and the development budget (W1/Q1).

We have clarified the contribution as a benchmark of autonomous AI-system development: selecting, implementing, and integrating methods into working systems under realistic resource constraints. Architecture, model size, optimizer, precision, and parallelism remain open in pretraining. Sharing a deadline between development and training tests the consequential decision of how much experimentation is worth its cost.

The hardware-budget comparison in Section 4.2 (p. 9)\color{blue}{\text{Section 4.2 (p. 9)}} and Appendix E.1 (pp. 20–21)\color{blue}{\text{Appendix E.1 (pp. 20–21)}} makes this decision-making concrete. In separate twelve-hour runs on two and four GB300 GPUs, Opus 5.5 chooses 20 and 24 layers, respectively, at the same hidden width of 1,536. The resulting models have 0.566B and 0.680B non-embedding parameters, train on 12.43B and 21.58B tokens, and reach PPLs of 13.783 and 12.769. Both trajectories show throughput experiments before model selection: the two-GPU run compares widths 1,536 and 2,048; the four-GPU run tests three widths and measures multi-GPU scaling. Main training starts at 39 and 44 minutes, respectively, after these experiments. The early start therefore follows empirical design choices within the time budget.

The revised Introduction (pp. 1–2)\color{blue}{\text{Introduction (pp. 1–2)}} and Figure 1 (p. 2)\color{blue}{\text{Figure 1 (p. 2)}} distinguish the motivating successor-development loop from the individual stages evaluated here. Repeated frontier-scale training and deployment cycles would require substantially greater resources; these tasks provide tractable measurements of the capabilities needed to undertake them.

Post-training now evaluates terminal-use agents: an agent builds the training system and improves a roughly 30B-parameter GLM-4.7-Flash model on four GB300s. Objectives, trainable parameter subsets, and rollout/training allocation are open choices. Self-distillation is permitted, including supervised training on the target model's own task-generated traces. None of the reported agents uses it; all choose policy-gradient RL. We make this scope explicit in Section 3.1 (pp. 3–6)\color{blue}{\text{Section 3.1 (pp. 3–6)}} and Appendix H.1 (p. 25)\color{blue}{\text{Appendix H.1 (p. 25)}}.

The restriction concerns external supervision. The task provides training environments and rewards, but no supervised solution traces. Allowing the authoring agent to supply its own answers as SFT targets would transfer its task-solving ability into the trained model's score. We treat this shortcut as reward hacking under the task's contract, so demonstrations generated by the authoring agent or another external teacher are prohibited. Self-distillation uses the target model's own experience.

The new midtraining data-curation task, integrated into the main benchmark (Section 3.1 (pp. 3–6)\color{blue}{\text{Section 3.1 (pp. 3–6)}}, Section 4.1 (pp. 8–9)\color{blue}{\text{Section 4.1 (pp. 8–9)}}, Table 4 (p. 8)\color{blue}{\text{Table 4 (p. 8)}}; Appendix G (pp. 23–24)\color{blue}{\text{Appendix G (pp. 23–24)}}), covers SFT after pretraining using supplied corpora. Agents select, filter, format, and mix data from 1.1 TB of corpora and may use continued pretraining, SFT, RL, or combinations. Every checkpoint then undergoes the same downstream RL stage. Together, the tasks cover both learning from curated demonstrations and improving a policy through its own experience.

2. Repeated runs (W2).

Appendix F (pp. 22–23)\color{blue}{\text{Appendix F (pp. 22–23)}} now reports three end-to-end pretraining runs for each of four open-weight models using Codex: Kimi K3, GLM 5.3, DeepSeek V4.1 Flash, and MiniMax M3. Run 1 is the result in Table 4 (p. 8)\color{blue}{\text{Table 4 (p. 8)}}; Runs 2 and 3 are fresh agent runs from the starting codebase and data, repeating design, implementation, training, and submission under the same twelve-hour, four-GB300 budget and provider-default reasoning effort (high).

Each cell below reports PPL / cross-entropy (authoring API cost). Cross-entropy is ln(PPL), in nats per token. Score means exclude failed runs; mean API cost includes all three runs.

Model + harnessRun 1Run 2Run 3Mean
Kimi + Codex14.431 / 2.6694 ($12.40)14.809 / 2.6953 ($14.49)17.838 / 2.8813 ($17.28)15.69 / 2.7487 ($14.72)
GLM + Codex16.311 / 2.7918 ($8.59)16.966 / 2.8312 ($7.14)16.603 / 2.8096 ($7.41)16.63 / 2.8109 ($7.71)
DeepSeek + Codex17.582 / 2.8669 ($0.54)16.015 / 2.7735 ($0.49)18.786 / 2.9331 ($0.50)17.46 / 2.8578 ($0.51)
MiniMax + CodexFail / — ($0.44)21.514 / 3.0687 ($3.69)18.257 / 2.9046 ($2.07)19.89 / 2.9867 ($2.07)

Both mean PPL and mean cross-entropy preserve the first-run ordering: Kimi, GLM, DeepSeek, then MiniMax, with failed submissions ranked last. MiniMax's first run ends without a checkpoint; its two successful repetitions still have the highest mean PPL and cross-entropy in this cohort.

The cross-entropy sample standard deviations are 0.1156, 0.0197, 0.0802, and 0.1160 nats/token, respectively, around mean losses of 2.7487, 2.8109, 2.8578, and 2.9867. The corresponding PPL standard deviations are 1.87, 0.33, 1.39, and 2.30. MiniMax's score means and standard deviations use its two successful runs. Reporting the individual runs and both scales quantifies variability while showing that the average performance ordering is consistent with the first-run comparison for these four configurations.

We plan further repetitions across the remaining agents and tasks. Limited compute availability currently prevents completing this broader evaluation.

3. Copying the installed harness or hard-coding answers (Q2).

The reviewer raises a good point, and we accounted for this by implementing a safeguard that prevents the agent from accessing the installed authoring harness. Command and file tools run under a separate worker identity whose filesystem permissions prevent reading or executing the private CLI installation. Installed coding-agent harnesses are also unavailable during downstream evaluation. The reported runs were therefore prevented from copying or invoking the installed harness to obtain baseline performance. We have made this explicit in Section 3.1 (pp. 3–6)\color{blue}{\text{Section 3.1 (pp. 3–6)}} and Appendix J.1 (p. 28)\color{blue}{\text{Appendix J.1 (p. 28)}}.

Downstream task identities and one of the three driver models are withheld during harness construction. The downstream verifiers include hidden correctness tests, and source and trajectory review checks for task-specific answer injection and unauthorized model access. General prompting strategies, such as requiring tests before completion, are legitimate harness design; embedding a particular evaluation task's solution is not.

4. Reasoning effort (Q3).

Section 4 (p. 7)\color{blue}{\text{Section 4 (p. 7)}}, Appendix A.3 (p. 17)\color{blue}{\text{Appendix A.3 (p. 17)}}, and Table 7 (p. 18)\color{blue}{\text{Table 7 (p. 18)}} now report authoring settings across all six tasks. Open-weight models use the provider default, confirmed to be high. Opus 5.5, Opus 5, and GPT-6.1 Sol use their strongest setting, max. These settings describe the authoring agents; the safeguard-validation judges use medium effort.

Reviewer_jK2wOpenReview ↗

Official Comment by Reviewer jK2w

Thank you for the detailed response and the revision. The reframing in the abstract, introduction, Figure 1 and conclusion addresses my first concern: described as a benchmark of autonomous AI-system development under fixed budgets, the tasks measure what they measure, and I find that contribution worthwhile. The reasoning-effort disclosure (Appendix A.3, Table 7) and the repeated pre-training runs (Appendix F) are appreciated.

On the harness task, the worker-identity isolation, hidden correctness tests and withheld task identities described in Section 3.1 and Appendix J.1 resolve the specific shortcuts I raised. I note that none of these protections appeared in the submitted version, so the red-team exercise reported in Section 3.2 did not cover them.

I also note for the record that the revised pre-training task runs on four GPUs rather than two, so every pre-training score and the human baseline in Table 4 differ from the submitted version, and the post-training task has been replaced. These are improvements, but a substantial part of the reported results is new in this revision. I keep my score.

Reviewer_XC9XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Singularity-Bench, a benchmark that tests whether AI agents can build the systems used to develop AI. It has five end-to-end tasks: pre-training a language model from scratch, post-training a search agent with RL, building an inference engine for a 320B MoE model, building a coding-agent harness, and managing a 28-model serving cluster. Each task gives the agent a small starting codebase and a fixed budget (4 to 12 hours, up to 16 GB300 GPUs). The agent's system is compared with a production system (Megatron-LM, slime, SGLang, Claude Code, Dynamo) on the same hardware. To prevent cheating, the authors add isolated testing, protected logs and two LLM judges; the judges caught all three red-team attacks in 5 out of 5 trials. The final score, the singularity index, is the average ratio to the baselines. Among 12 agents, Opus 5.5 + Claude Code beats the baseline on all five tasks (index 1.275). The other agents average 0.56, and 8 of 12 score zero on inference.

Critical strengths
  1. The tasks are realistic and open-ended. Many benchmarks fix most of the system or use small single-GPU settings. Here, agents must choose the architecture, optimizer, parallelism and scheduling themselves, at a realistic scale and across the whole development stack.
  2. The anti-cheating work and the failure analysis are useful. Red-teaming led to concrete safeguards. The DeepSeek + Codex violation left no trace in the submitted harness and was visible only in the development trajectory, which shows that the process must be checked, not only the final result. The appendices report all 60 audit results and explain common failures in detail, such as evaluation bugs at batch boundaries, sessions that end while expecting background jobs to continue, and fixed turn limits.
Critical weaknesses
  1. The singularity claim goes beyond what is measured. Figure 2 defines singularity as an agent that can improve every stage of its own development, again and again. But the five tasks are tested separately and at small scale (e.g., a 1B model trained for 12 hours), and no agent's output is used in the next stage. Important stages such as data curation are not covered. Appendix A.1 itself says that testing parts separately does not show an agent can combine them into a stronger successor. Still, the abstract, Figure 1, the introduction and Section 4 say Opus 5.5 might already reach singularity, while the conclusion only describes "substantial but uneven progress." The claims should match the evidence.
  2. The comparison with the baselines is weaker than it looks. The "human baselines" are public recipes that were not tuned for these budgets. For pre-training, only model size was tuned, and the baseline lowers its learning rate to 10% instead of zero, which Table 6 lists as one of Opus 5.5's advantages. In post-training, 8 of 12 agents beat the baseline (0.253), which suggests the baseline is weak. Each setup is run only once, with no repeated seeds and different numbers of restarts. Opus 5.5's harness lead (0.468 vs. 0.4375) comes from one extra success out of 15; without it, the agent would be below the baseline. The costs also leave out GPU time and failed attempts, so the cost comparison is incomplete.
Questions for authors
  1. Could you add a real human baseline for at least one task, with experienced engineers working under the same time, hardware and offline limits? If not, how do the results change when the current baselines are tuned for the task budgets?
  2. How will the benchmark stay difficult? The baselines are fixed, the tasks are public, and one agent already beats the baseline on all five tasks, so the human-baseline reference may soon stop being informative. Do you plan to update the baselines, change the target models, or chain the tasks so that an agent's own model passes through its own systems?
5 comments

Thank you for highlighting the realistic, open-ended tasks and the value of the safeguards and failure analysis. The revised paper evaluates eleven configurations across six tasks, adds data-curation results, and replaces the search-RL experiment with terminal-agent post-training against Miles. Changes are highlighted in blue.

1. What the benchmark establishes, and data curation (W1).

We fully understand this concern and have revised the paper extensively to make its claims more rigorous. Testing whether an agent can combine every stage into a stronger successor would require repeated frontier-scale training and deployment cycles, with substantially greater resources than a fixed-budget benchmark can provide. We have therefore rewritten the Abstract (p. 1)\color{blue}{\text{Abstract (p. 1)}}, Introduction (pp. 1–2)\color{blue}{\text{Introduction (pp. 1–2)}}, Figure 1 (p. 2)\color{blue}{\text{Figure 1 (p. 2)}}, and Conclusion (p. 10)\color{blue}{\text{Conclusion (p. 10)}} to consistently describe the benchmark as measuring, under fixed resource budgets, the skills required at each stage of the end-to-end development of a next-generation agent. Because the tasks use realistic workloads and are compared against production-grade baselines, they serve as a strong proxy for the end-to-end engineering skills required in industry-scale LLM development. We also agree that data curation is a key stage and have added it as a main task, described below.

Data curation is now a sixth main task, with results in Section 4.1 (pp. 8–9)\color{blue}{\text{Section 4.1 (pp. 8–9)}} and Table 4 (p. 8)\color{blue}{\text{Table 4 (p. 8)}}. We add a mid-training task that specializes on cleaning, deduplicating, and curating data mixtures for mid-training. Agents select, filter, deduplicate, mix, and format data from 1.1 TB of supplied corpora, then mid-train Llama-3.2-1B for twelve hours on four GB300s. Every checkpoint subsequently undergoes the same five-hour RL stage and evaluation on mathematics, code, and science. Opus 5.5 reaches mean accuracy 0.3188, compared with 0.1755 for PRISM (the human baseline), 0.1175 for uniform mixing, and 0.0953 for RL without midtraining. Appendix G (pp. 23–24)\color{blue}{\text{Appendix G (pp. 23–24)}} details the controls and curation decisions. This directly tests whether agent-selected data and training improve subsequent learning. We believe the inclusion of this new task strengthen the data curation portion of agent development as the reviewer suggested.

2. Baseline strength and matched human comparisons (W2/Q1).

The baselines incorporate months to years of human research and engineering by expert teams and large communities, including optimized kernels, distributed systems, and training recipes. Agents must build their solutions from minimal starting codebases within four to twelve hours. Section 3.1 (pp. 3–6)\color{blue}{\text{Section 3.1 (pp. 3–6)}} now makes this context and the task-specific baseline choices explicit:

  • Pretraining: baseline preparation included comparing 1B and 7B models and tuning the parallelism strategy, mixed-precision configuration, and learning rate schedule for OLMo 2-1B in Megatron-LM on four GB300s. The best configuration tested uses FP8; for learning rate schedule, we compared cosine decay to 10% (PPL 14.216), cosine decay to zero (14.280), and OLMo 2's two-stage schedule scaled to twelve hours (14.171). We select the two-stage schedule: cosine decay to 10% over the first 98.7% of tokens, then linear decay to zero over the remaining 1.3%. The baseline's full twelve hours are devoted to training. Appendix E.3 (p. 22)\color{blue}{\text{Appendix E.3 (p. 22)}} documents the tuning and selection.
  • Inference: SGLang is a state-of-the-art open-source serving system. We use v0.5.20 with its published GLM-5.3-Flash recipe. Agents know the workload specification, but the exact evaluation traffic is selected with a hidden seed and unavailable in advance.
  • Harness: Claude Code is a leading production coding-agent harness, evaluated on the same five tasks with the same three driver models.
  • Cluster: NVIDIA Dynamo with KAI is a strong open-source orchestration baseline. Dynamo has been benchmarked on GB300 systems, including NVIDIA's published MLPerf results.
  • Midtraining: PRISM is a strong published data-mixture and training recipe, recognized as an ICML 2026 Spotlight. It uses the same midtraining budget and downstream RL procedure as the agents.

The new post-training experiment further strengthens the comparison. Agents build an RL system for the approximately 30B-parameter GLM-4.7-Flash model, evaluated on held-out terminal and SWE-bench tasks. The human baseline is Miles' full-parameter GRPO recipe, adapted to the same four GB300s, twelve-hour budget, and agent scaffold, with its auxiliary MTP loss disabled to match the rules. It scores 0.5388, versus 0.5332 for Opus 5.5, 0.5301 for GPT-6.1 Sol, and 0.5175 for Opus 5; the untrained model scores 0.3842. The leading agents substantially improve the policy and approach this strong human-built training system (Section 3.1 (pp. 3–6)\color{blue}{\text{Section 3.1 (pp. 3–6)}}; Appendix H (pp. 25–26)\color{blue}{\text{Appendix H (pp. 25–26)}}).

For the proposed twelve-hour human study, matching the task also means matching the starting code, library access, and time spent implementing and debugging. An expert would need to build and test a working system while leaving enough time for training. Our human-built baselines already enter the timed runs with complete, configured Megatron-LM or Miles training systems. Megatron-LM and Miles were unavailable to the agents in the reported runs. Agents begin from nanoGPT or a minimal synchronous GRPO trainer and must develop comparable training capabilities within the deadline. This gives the baselines a substantial head start in prior engineering investment, making competitive agent performance meaningful. A matched expert study would additionally quantify development speed under identical starting conditions; it cannot be completed with the available rebuttal resources. Section 3.1 (pp. 3–6)\color{blue}{\text{Section 3.1 (pp. 3–6)}} and Appendix A.6 (pp. 18–19)\color{blue}{\text{Appendix A.6 (pp. 18–19)}} now explain this distinction.

3. Variance, recovery runs, and the harness margin (W2).

Appendix F (pp. 22–23)\color{blue}{\text{Appendix F (pp. 22–23)}} now reports three end-to-end pretraining runs for each of four open-weight models using Codex: Kimi K3, GLM 5.3, DeepSeek V4.1 Flash, and MiniMax M3. Run 1 is the result in Table 4 (p. 8)\color{blue}{\text{Table 4 (p. 8)}}; Runs 2 and 3 are fresh agent runs from the starting codebase and data, repeating design, implementation, training, and submission under the same twelve-hour, four-GB300 budget and provider-default reasoning effort (high).

Each cell below reports PPL / cross-entropy (authoring API cost). Cross-entropy is ln(PPL), in nats per token. Score means exclude failed runs; mean API cost includes all three runs.

Model + harnessRun 1Run 2Run 3Mean
Kimi + Codex14.431 / 2.6694 ($12.40)14.809 / 2.6953 ($14.49)17.838 / 2.8813 ($17.28)15.69 / 2.7487 ($14.72)
GLM + Codex16.311 / 2.7918 ($8.59)16.966 / 2.8312 ($7.14)16.603 / 2.8096 ($7.41)16.63 / 2.8109 ($7.71)
DeepSeek + Codex17.582 / 2.8669 ($0.54)16.015 / 2.7735 ($0.49)18.786 / 2.9331 ($0.50)17.46 / 2.8578 ($0.51)
MiniMax + CodexFail / — ($0.44)21.514 / 3.0687 ($3.69)18.257 / 2.9046 ($2.07)19.89 / 2.9867 ($2.07)

Both mean PPL and mean cross-entropy preserve the first-run ordering: Kimi, GLM, DeepSeek, then MiniMax, with failed submissions ranked last. MiniMax's first run ends without a checkpoint; its two successful repetitions still have the highest mean PPL and cross-entropy in this cohort.

The cross-entropy sample standard deviations are 0.1156, 0.0197, 0.0802, and 0.1160 nats/token, respectively, around mean losses of 2.7487, 2.8109, 2.8578, and 2.9867. The corresponding PPL standard deviations are 1.87, 0.33, 1.39, and 2.30. MiniMax's score means and standard deviations use its two successful runs. Reporting the individual runs and both scales quantifies variability while showing that the average performance ordering is consistent with the first-run comparison for these four configurations.

We plan further repetitions across the remaining agents and tasks. Limited compute availability currently prevents completing this broader evaluation.

Appendix A.2 (p. 17)\color{blue}{\text{Appendix A.2 (p. 17)}} distinguishes recovery after task/infrastructure faults from fresh repetitions and from an agent's own training restarts within its budget.

For harness construction, the claim is competitive performance with Claude Code. Section 4.1 (pp. 8–9)\color{blue}{\text{Section 4.1 (pp. 8–9)}} and Appendix J (pp. 28–30)\color{blue}{\text{Appendix J (pp. 28–30)}} explicitly identify the extra MiniMax-driven PR success behind the 0.468 versus 0.4375 comparison. Each binary success contributes 1/15 to the score; the task/driver decomposition makes this margin inspectable without treating it as an established statistical advantage.

4. Cost accounting (W2).

Table 4 (p. 8)\color{blue}{\text{Table 4 (p. 8)}} labels costs as authoring API spend; Appendix A.4 (pp. 17–18)\color{blue}{\text{Appendix A.4 (pp. 17–18)}} separates GPU allocation, evaluation, and other attempts. Pretraining authoring costs total $142.28 for the main-table runs and $189.84 for other recorded attempts. Runs 2 and 3 in Appendix F (pp. 22–23)\color{blue}{\text{Appendix F (pp. 22–23)}} add $53.07 in authoring API spend. Reported midtraining and post-training runs total $152.46 and $222.61, respectively. Nominal development allocations are 48 GPU-hours per pretraining, midtraining, post-training, or inference run and 64 GPU-hours per cluster run. Midtraining's common downstream RL stage adds 20 GPU-hours per checkpoint; verification and downstream harness execution are additional. These figures make the API-cost scope and compute budgets explicit.

5. Keeping the benchmark informative (Q2).

The benchmark is far from saturated. As Table 4 (p. 8)\color{blue}{\text{Table 4 (p. 8)}} shows, raw scores on each task remain low, leaving substantial room for improvement. Six of eleven configurations fail the inference task, and no agent beats Miles on the updated post-training task. Scores are also not capped at the human baselines, so agents can continue improving beyond them. The baseline scores are only served to ground the top-agent performance and produce a shared normalized metrics across all six tasks. When stronger human-built systems become openly available, we will update the baselines accordingly, and we will continue to evaluate new and stronger agents as they are released (Appendix A.6 (pp. 18–19)\color{blue}{\text{Appendix A.6 (pp. 18–19)}}).

Reviewer_So4ROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Singularity-Bench, a benchmark of five open-ended tasks covering the stack used to build LLM agents: pre-training a language model from scratch, building search-augmented RL post-training, building an FP8 serving engine for GLM-5.3-Flash, building a coding-agent harness, and orchestrating serving for 28 models on a 16-GPU cluster. Each task gives the agent a minimal codebase, a fixed budget of 4 to 12 hours on GB300 GPUs and a single objective, and compares the result with a production, human-built reference on the same hardware. An extended Harbor framework isolates test execution, keeps protected logs and has two LLM judges audit each run. The Singularity Index averages each agent's scores relative to the references. Across 12 agents, Opus 5.5 with Claude Code exceeds the references on all five tasks (index 1.28), while the other agents average 0.56 and 8 of 12 score zero on inference.

Critical strengths

(++) The tasks are motivated by real-world LLM development practice. They run on real hardware against production systems with realistic workloads, such as replayed coding-agent traffic for serving.

(++) The evaluation is specifically designed to resist cheating: isolated test execution, protected logs and two independent judges caught all three attacks in a red-team test, and the audit of all 60 final runs is reported openly.

Critical weaknesses

(---) The singularity framing overstates what the benchmark measures. Describing it as a full-stack LLM engineering benchmark under fixed budgets would match the evidence. Data collection and curation play a key part in LLM training, and arguably in any path to singularity, yet every task supplies its data.

(---) The references are production recipes run with their published configurations (only model size was swept, for pre-training). They were not optimized by human engineers with the same hardware and time. So an index above 1 shows that an agent beats untuned default solutions, far from the abstract's suggestion that Opus 5.5 "might already reach singularity".

(--) The wins in Table 6 come mostly from known techniques that a model with a recent training cutoff can be expected to propose and implement, such as sizing the model to the budget, Muon in place of AdamW and a learning rate decayed to zero in pre-training.

(--) Each agent–task pair is reported from a single run, so small gaps between agents have no variance estimate.

(-) Each task is scored by one metric on one workload, while real systems must balance several. For example, the serving engine is scored on income per GPU-hour over 16 replayed sessions. A system specialized to one workload is expected to beat a general-purpose one on that workload, but it doesn't mean it's more useful in the real world.

Questions for authors
  • How many attempts did each agent get per task? How was the retained run chosen given the unequal restart histories? Was it simply the last run?
  • Have you observed parts of the winning solutions that go beyond published designs? Highlighting them would strengthen the case that agents contribute something new.
  • Averaging the five ratios with equal weight is not fair to tasks of different difficulty. For example, Opus 5.5's post-training ratio (1.59) is much larger than its pre-training ratio (1.20), yet the 16% drop in perplexity may be the harder gain, since exact-match accuracy starts from a low reference (0.253). Reporting per-task scores as the main result would be easier to interpret.
6 comments

Thank you for recognizing the realistic workloads and evaluation safeguards. We have revised the paper in blue, with eleven configurations evaluated on six tasks, new data-curation and terminal-agent post-training results, and per-task reporting throughout.

1. Framing and data curation (W1).

We agree that the singularity framing overstated what the benchmark measures. Evaluating the full successor-development loop, with repeated frontier-scale training and deployment cycles, would require substantially greater resources than a fixed-budget benchmark can provide. Following your suggestion, we have rewritten the Abstract (p. 1)\color{blue}{\text{Abstract (p. 1)}}, Introduction (pp. 1–2)\color{blue}{\text{Introduction (pp. 1–2)}}, Figure 1 (p. 2)\color{blue}{\text{Figure 1 (p. 2)}}, and Conclusion (p. 10)\color{blue}{\text{Conclusion (p. 10)}} to describe the benchmark as measuring autonomous AI-system development under fixed resource budgets. The successor-development loop now serves only as motivation: each task measures a capability required by one stage of that loop, providing tractable measurements of progress toward it rather than a test of the loop as a whole. Because the tasks use realistic workloads and are compared against production-grade baselines, they serve as a strong proxy for the end-to-end engineering skills required in industry-scale LLM development. We also agree that data curation is central and have added it as a main task.

Midtraining data curation is now a main benchmark task (Section 3.1 (pp. 3–6)\color{blue}{\text{Section 3.1 (pp. 3–6)}}, Section 4.1 (pp. 8–9)\color{blue}{\text{Section 4.1 (pp. 8–9)}}, Table 4 (p. 8)\color{blue}{\text{Table 4 (p. 8)}}; Appendix G (pp. 23–24)\color{blue}{\text{Appendix G (pp. 23–24)}}). Agents choose data from 1.1 TB of supplied corpora and control filtering, deduplication, mixtures, formatting, curriculum, and training method. Following twelve hours on four GB300s, each checkpoint undergoes the same five-hour RL stage and evaluation on mathematics, code, and science. Mean accuracy is 0.3188 for Opus 5.5, 0.1755 for PRISM (the human baseline), 0.1175 for uniform mixing, and 0.0953 for RL without midtraining. These results make the consequences of curation and preparation for further learning directly measurable.

2. Human baselines (W2).

The baselines incorporate months to years of human research and engineering by expert teams and large communities. This accumulated work includes training recipes, optimized kernels, and production infrastructure. In contrast, agents must build their solutions from minimal codebases within four to twelve hours. Section 3.1 (pp. 3–6)\color{blue}{\text{Section 3.1 (pp. 3–6)}} now makes the baseline choices and configuration work explicit:

  • Pretraining: baseline preparation included comparing 1B and 7B models and tuning the parallelism strategy, mixed-precision configuration, and learning rate schedule for OLMo 2-1B in Megatron-LM on four GB300s. The best configuration tested uses FP8. We compared three learning-rate schedules under the same twelve-hour training budget:

    Learning-rate schedulePPL ↓
    Cosine decay to 10% of peak14.216
    Cosine decay to zero14.280
    Cosine decay to 10% over the first 31.0B tokens (98.7%), then linear decay to zero over the final 424M tokens (1.3%)14.171

The original Olmo recipe describes a two-stage learning-rate schedule: the first 98.7% of the training corpus uses cosine decay to 10%, and the remaining <2% of training tokens decay linearly to 0. In our FP8 training of 4xGB300 for 12 hours, we are able to train more than 32B tokens. Our original baseline in the paper is based on the first-stage training alone. Since the comments, we tried the two alternative designs the reviewer is interested in: decaying learning directly to zero, and following the same proportion as the original recipe and decaying it in two stages. We show it in the above table, and all three are numerically similar, which fundamentally cannot out-compete the sub-13 PPL submission delivered by Opus 5.5+CC.

We select the two-stage schedule as the new baseline score, as it is marginally the lowest. The baseline's full twelve hours are devoted to training. Appendix E.3 (p. 22)\color{blue}{\text{Appendix E.3 (p. 22)}} documents the tuning and selection.

  • Inference: SGLang is a state-of-the-art open-source serving system; we use v0.5.20 with its published GLM-5.3-Flash recipe. Agents know the workload specification but do not receive the exact evaluation traffic in advance: the sessions are selected with a hidden seed.
  • Harness: Claude Code is a leading production coding-agent harness, evaluated on the same five tasks with the same three driver models.
  • Cluster: NVIDIA Dynamo with KAI is a strong open-source orchestration baseline, and NVIDIA reports Dynamo performance on GB300 systems.
  • Midtraining: PRISM provides a strong published data-mixture and training recipe, recognized as an ICML 2026 Spotlight. Its hardware, midtraining budget, and downstream RL procedure match the agent comparison.

The revised post-training experiment provides an additional strong comparison: agents build an RL system for the approximately 30B-parameter GLM-4.7-Flash model and compete with Miles' full-parameter GRPO recipe, adapted to the same hardware, twelve-hour budget, and agent scaffold, with its auxiliary multi-token-prediction loss disabled. Miles scores 0.5388, compared with 0.5332 for Opus 5.5, 0.5301 for GPT-6.1 Sol, and 0.5175 for Opus 5; the untrained model scores 0.3842 (Appendix H (pp. 25–26)\color{blue}{\text{Appendix H (pp. 25–26)}}). The leading agents substantially improve the policy and approach the human-built recipe.

These comparisons give the human-built baselines a substantial head start in prior engineering investment: the training baselines enter the timed runs with complete, configured Megatron-LM or Miles systems, whereas agents begin from nanoGPT or a minimal synchronous GRPO trainer. Megatron-LM and Miles were unavailable to the agents in the reported runs. Agents must develop comparable training capabilities and fit implementation, debugging, experimentation, and training into twelve hours. A twelve-hour human comparison would also need to hold the starting code and library access fixed; it would measure development speed as well as final solution quality. Section 3.1 (pp. 3–6)\color{blue}{\text{Section 3.1 (pp. 3–6)}} and Appendix A.6 (pp. 18–19)\color{blue}{\text{Appendix A.6 (pp. 18–19)}} now explain how that experiment complements comparison against mature human-built systems. We retain the index as a descriptive summary of the measured comparisons, as explained in response 7.

3. Known techniques and contributions beyond published designs (W3/Q2).

The strongest evidence concerns autonomous selection, adaptation, and integration into complete systems. Section 4.2 (p. 9)\color{blue}{\text{Section 4.2 (p. 9)}} and Table 5 (p. 9)\color{blue}{\text{Table 5 (p. 9)}} highlight decisions supported by the trajectories: Opus 5.5 sizes its pretraining model from measured throughput and schedules learning-rate decay against the deadline; its new post-training system trains a 1.52B-parameter subset with asynchronous rollouts and approximately 0.8-second partial-weight pushes; its inference engine combines recurrent-state prefix caching, exact speculative decoding, and admission scheduling; and its cluster policy uses measured GPU interference to coordinate colocated engines.

We add a targeted case study of principled selection by Opus 5.5 (Section 4.2 (p. 9)\color{blue}{\text{Section 4.2 (p. 9)}}; Appendix E.1 (pp. 20–21)\color{blue}{\text{Appendix E.1 (pp. 20–21)}}). This experiment demonstrates that the top agent makes careful, well-grounded choices among established techniques, helping them achieve strong results. We run Opus 5.5 on two different and independent nano_openend_gpt task settings: two and four GB300s, both twelve-hour pretraining runs, varying only the hardware budget. Opus 5.5 adapts the different constraint and modify its design according to each budget: at the same width of 1,536, it chooses 20 versus 24 layers (0.566B versus 0.680B non-embedding parameters) and trains on 12.43B versus 21.58B tokens, reaching PPLs of 13.783 versus 12.769. The trajectories show that these choices follow measurement: before committing, the agent runs throughput probes, compares model widths, and measures four-GPU scaling. This case study shows the agent measures its environment and then selects model capacity and training volume to fit the resource budget, which is a form of engineering judgment that goes beyond simply applying a familiar recipe.

In end-to-end AI-system development, this ability to plan and coordinate every step around hard constraints is itself a critical skill. Knowing a technique is not sufficient, as an agent must decide which techniques to use, how to size and schedule them, and how to integrate them under a fixed compute and time budget. This is especially important as errors at any step compound across the pipeline, so such decisions are often the determining factor in whether a system succeeds. However, existing benchmarks rarely measure this skill, which is why we treat it as a central target of evaluation. We therefore identify the agents' contributions at this level of engineering judgment, without claiming priority for a new learning algorithm. The analysis also distinguishes observed design decisions and development measurements from effects that would require ablations.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Summary for AC and a Request for AC's Discretion

Dear Area Chair,

We sincerely thank you and Reviewers So4R, XC9X, and jK2w for the time and care invested in reading Singularity-Bench. The reviews were constructive and made the paper substantially stronger. The reviewers raised four main concerns, each of which we have addressed in the revision (changes in blue):

  • Framing and scope (all reviewers). The singularity claim went beyond what the benchmark measures, and data curation was missing. We rewrote the abstract, introduction, Figure 1, and conclusion to describe a benchmark of autonomous AI-system development under fixed budgets. We also added mid-training data curation as a sixth main task, where Opus 5.5 reaches 0.3188 against 0.1755 for the PRISM baseline (Table 4, Appendix G).
  • Strength of human baselines (So4R, XC9X). We tuned the pre-training baseline over model size, parallelism, precision, and three learning-rate schedules (PPL 14.171–14.280, Table 13), all well above the 12.769 reached by Opus 5.5. We also replaced the post-training task with a harder one against Miles' full-parameter GRPO recipe, which no agent beats (0.5388 against 0.5332 for Opus 5.5, Table 16).
  • Variance and reporting (all reviewers). We added three fresh end-to-end pre-training runs for each of four configurations, whose mean ordering matches the first-run ordering (Table 14, Appendix F). We also now disclose reasoning effort, recovery runs, and cost scope (Appendix A.2–A.4), and report per-task scores as the main result, with the index's limitations stated (Section 3.3).
  • Evaluation integrity (jK2w, XC9X). The harness task isolates the installed authoring harness and uses hidden correctness tests and withheld task identities (Section 3.1, Appendix J.1). Red-teaming now covers all six tasks, including the revised ones, across nine of ten OWASP Agentic Top 10 categories, with 22 attack scenarios plus six judge-aware ones, all caught (Tables 8–9, Appendix B).

Score changes. Reviewer So4R raised their rating from 2 to 3 (weak acceptance), stating that the revision "addresses most of my concerns." Reviewer jK2w kept 3 (weak acceptance), confirming that their first concern and the harness shortcuts are resolved and that they "find that contribution worthwhile." Reviewer XC9X kept 2 (weak rejection), and we respectfully ask the AC to weigh this score against the reviewer's own assessment. XC9X states that "the revised framing and the stronger pre-/post-training baselines address my main concerns," and none of the weaknesses in their original review remains in dispute. The reasons given for keeping the score are new to the discussion, and we believe each is answered by evidence in the revised paper:

  1. The new evidence "has not been red-teamed." It has: Tables 8–9 red-team all six tasks, including the updated pre-training, post-training, and mid-training tasks.
  2. The mid-training gain may reflect output formatting. The RL stage and the evaluation use the same answer-extraction functions, so evaluation adds no new format requirement. Opus 5.5's advantage already appears in RL training reward (0.262 against 0.190, Table 15). On HumanEval+, both models emit fenced code in about 87% of outputs, yet Opus 5.5 passes 71/164 execution tests against 39/164 for PRISM.
  3. Much of the evidence is new. These experiments were added in direct response to the reviewers' requests for data curation, stronger baselines, and repeated runs. We hope that being responsive within the discussion period is not counted against the paper.

XC9X's comment closed with "No further response is needed," and our reply of 18 Nov with the evidence above has not received a response. We therefore believe the retained score does not reflect the current state of the paper.

Request. We kindly ask for the AC's judgement in light of the above. We believe the paper makes three contributions of value to the community:

  • A full-stack, open-ended benchmark. Six end-to-end tasks cover pre-training, mid-training data curation, post-training, inference, the agent harness, and cluster orchestration. Agents start from minimal codebases on real hardware at realistic scale and are compared against mature human-engineered systems (Megatron-LM, PRISM, Miles, SGLang, Claude Code, NVIDIA Dynamo).
  • A defense-in-depth evaluation framework. Isolated execution, protected logging, and independent judges are validated by red-teaming across all tasks. All three reviewers listed this as a strength.
  • A clear empirical picture of 11 agents. Progress is substantial but highly uneven: the best agent builds an inference engine earning 41% more per GPU-hour than SGLang and outperforms PRISM on mid-training, while six of eleven agents score zero on inference and no agent beats Miles on post-training. The benchmark is therefore far from saturated.

Thank you again for your time and consideration.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.