acceptodds
Withdrawn from ICLR 2027

Stellar Colosseum: A Many-Agent Harness for Long-Horizon Research in Mathematics and Theoretical Computer Science

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language models can produce plausible short proofs, but may still be unreliable on long-horizon research problems, where progress depends on a sequence of uncertain and interdependent decisions. We introduce Stellar Colosseum, a model-agnostic harness for allocating inference across research in mathematics and theoretical computer science. Colosseum explores alternative strategies before proof construction, uses a readiness gate to decide when a route is mature enough to decompose, represents the proof plan as interdependent section-level subproblems, and routes verifier findings back to the affected part of the argument. Across these stages, it generates candidates in parallel, attacks them with targeted falsification, and combines candidates and their critiques into a single research artifact through overlapping random-sample tree aggregation. We demonstrate the capabilities of Colosseum through open-ended research and evaluations on theorem-proving and competitive programming benchmarks. Using Colosseum with Gemini 3.1 Pro, we obtain several new results that address open problems arising from papers published at top venues such as FOCS and JMLR. On TCS-Bench (Cohen-Addad et al., 2026), a benchmark of research-level theorem-proving tasks drawn from papers published at FOCS, STOC, and SODA, Colosseum achieves 71.0% accuracy using Gemini 3.1 Pro and Gemini 3.7 Flash. In a separate Codeforces evaluation using Gemini 3.1 Pro, the proof-oriented pipeline with execution feedback solves 218 of 222 problems.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM agents, automated research, long-horizon reasoning, multi-agent collaboration, tree-structured aggregation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SNfgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Stellar Colosseum is a many-agent workflow for research in mathematics and theoretical computer science. It explores strategies before building a proof, uses a readiness gate before decomposition, and repairs sections using their dependency graph. Within each stage, overlapping groups of candidates and critiques are synthesized into a final output. The paper also analyzes evidence retention in an idealized aggregation DAG. Evaluation includes five research contributions, two case studies, TCS-Bench, and Codeforces. It reports 71.0% on TCS-Bench with cross-model selection and 218/222 on Codeforces with execution feedback.

Critical strengths
  1. Keeping critiques attached to candidates is a useful design choice. A concrete objection can survive aggregation even when most candidates miss the same gap. The dependency graph and local repair also suit long proofs. Theorem 1 gives a clear analysis of evidence retention under explicit assumptions, including the logarithmic regime at the critical threshold.
  2. On TCS-Bench, Gemini 3.1 Pro improves from 30.3% to 54.0% with the harness (Table 3). The Codeforces evaluation checks accepted programs against the complete hidden tests using the original checkers. I also give weight to the coreset improvement and the long proof drafts for Knuth's cycles. These provide evidence from research tasks beyond the benchmarks.
Critical weaknesses
  1. It is hard to separate the effect of the design from the larger inference budget. Table 1 gives tree widths, but not actual token use, runtime, or retry cost. The 71.0% result requires two full harness runs and eight additional critiques. There is no baseline using repeated sampling and verification at the same budget, nor an ablation of overlap or critique retention. The Codeforces comparison isolates execution feedback only. Appendix G.1 acknowledges these gaps. The retention theorem studies evidence survival under independence assumptions; it does not establish which components improve proof success.
  2. I would like more detail on validation and human involvement. Section 5.1 states that the authors of the cited papers checked the mathematics, but neither that section nor Appendix D gives a run-level record of interventions or corrections. The unit-distance example is presented as strategy rediscovery; the verification status of its 22-page draft remains unclear. TCS-Bench relies on an automated grader. Its reported accuracy above 90% on 100 expert-labeled proofs does not establish its false-acceptance rate on the harness outputs. The AUC of 0.896 also uses this grader's labels.
Questions for authors
  1. What were the token use and wall time per problem for the Gemini 3.1 Pro TCS-Bench run, including verification and retries? A useful additional experiment would compare with repeated sampling and verification at the same budget. Could you also test one ablation of overlap or critique retention? A representative subset would help.
  2. Was the grader's reported accuracy above 90% measured on held-out examples or on the 100 proofs used to tune its prompt? Has an expert audit checked final harness proofs, especially cases where the internal verifier and cross-model selector disagree? If so, please report the audit size and the numbers of false acceptances and false rejections.
  3. For one result in Table 2, could you share the initial task, unedited harness output, and a short log of human interventions and later corrections? For the unit-distance draft, which claims have been independently checked and which remain unverified?
Reviewer_TuVROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a multi-agent harness for long-horizon mathematical research. It combines strategy exploration, a readiness gate, dependency-aware proof construction, local revision, and global verification. Within each stage, it uses overlapping tree-structured aggregation to synthesize candidates while carrying their associated falsification reports forward. The paper also provides an idealized analysis of evidence survival and reports strong results on TCS-BENCH, competitive programming, and several research case studies.

Critical strengths

The paper tackles an important difficulty: long proofs require not only generating ideas, but also tracking dependencies, revisiting failed approaches, and preventing unresolved objections from disappearing during synthesis. Recent works such as Danus have placed significant efforts onto such difficulties. The proposed workflow is coherent, and the reported end-to-end results show that the full system is capable of producing substantial outputs.

Critical weaknesses

My concern is that the paper does not effectively establish that its distinctive design choices are responsible for those results provided in the paper. The most specific methodological proposal appears to be overlapping aggregation that preserves candidate-linked falsification reports, embedded in a workflow with readiness gating, dependency-aware decomposition, and local retries. However, the evaluation largely treats this collection of mechanisms as a single package. This distinction matters because the system uses substantial inference-time computation. The theoretical result does not close this gap. It characterizes when an item of evidence survives an idealized aggregation process under assumed sampling and retention behavior. It does not establish that real LLM aggregators actually improve final proof correctness. Thus, the theorem cannot fully support the paper’s central empirical claim about the utility of this particular design.

Questions for authors
  1. At a fixed inference budget, how much of the performance gain remains relative to a simpler generate–verify–revise baseline?Can the authors provide a compute-matched end-to-end comparison against a simple generate–verify–revise system and boN selection, reporting proof quality as a function of total inference cost?

  2. What is the marginal effect of carrying falsification reports through aggregation, separately from the effect of overlap and additional computation?

  3. Besides, it is possible to provide an ablation of the central aggregation mechanism?

Reviewer_qudcOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work presents Stellar Colosseum, a model-agnostic harness for creating long-horizon mathematical proofs. The presented architecture consists of (1) devising a proof strategy, (2) a readiness gate to verify the strategy's concreteness, (3) a proof decomposer to break it down into subgoals, (4) solver stage, where the subparts are resolved in parallel, and (5) a global verification stage. When a verifier (either the gate or global verifier) rejects the result, the harness is allowed to go back and retry from where the failure seems to stem. Each stage is guided by a wide search, where candidates are falsified to provide feedback for potentially unclear or untrue items, and aggregated to produce a single full claim. The whole system yields a single final proof. Colosseum was evaluated on 3 aspects across 1--2 Gemini models on resolving open problems, theoretical computer sceince, and programming. It contributed to 5 open problems, present reasonable advantage over base models in TCS, and is able to resolve nearly all programming challenges.

Critical strengths
  1. The steps in which the framework is broken down make intuitive sense and match well the process of research, including introspection, diverse attempts, and retrying upon noticing failure.

  2. The open problem results seem impressive, though only 1 is presented in the submission.

Critical weaknesses
  1. Poor motivation -> The paper make little if any attempt to explain what advantage Colosseum carries over any other agentic framework, besides introducing new components, and in turn does not motivate where the need for these components stems from. It also does not address what shortcomings it aims to tackle. In particular, it remains unclear what necessitates yet another agentic system, or in what way Colosseum addresses its long-horizon claim, which features prominently in the title and abstract. For instance, ProofCouncil, which the paper has cited, also contains some similar components, i.e. an author-critic loop and multiple parallel iterations. Hence, what issues the additions of Colosseum address remains unresolved, which is largely a presentation issue. These concerns carry over to the evaluation of the framework, as described below.

  2. Significantly poor evaluation -> Overall, the paper makes several evaluations to establish Colosseum's worth. However, I find that the evaluation section lacks the sufficient depth to establish several of the paper's most important claims. I have classified those issues separately below, and include what I believe to be the degree of importance of those concerns.

    • (Major) No baselines in 2/3 evaluations: 2/3 of the evaluation sections feature no baselines. For 5.1, the showcased results are mostly case studies, so that is at least acceptable, but it still remains unclear whether any other harness than Colosseum could have achieved the same results (i.e. a simple Claude Code/Codex instance). For the CodeForces showcase, without any baselines it can only serve as an ablation on the execution probe, which is marginal value at best, and provides no evidence of the value of the harness as a whole.
    • (Major) No agentic baselines: In the remaining Section 5.2, while there are some base-model comparisons available (though also incomplete, lacking 3.7-Flash), there are no comparisons to other agentic harnesses. For example, it remains unclear whether a ClaudeCode, Codex, or any other generalist harnesses cannot achieve similar results, invalidating the need for a math-specific one. Even if that were established, whether Colosseum outperforms math-specific harnesses, i.e. ProofCouncil, remains open. If such a study were to be performed, it also need be compute-matched, as described in the point below. Overall, any advantage Colosseum's structure provides over simple test-time compute scaling, even simple best-of-n selection remains unclear.
    • (Major--Medium) No cost information: No compute cost was provided, in terms of API credits (more important) or runtime. This is important, especially if the harness turns out to be significantly more expensive than similarly effective alternatives. While the authors acknowledge the lack of such information, it does not excuse them for not including it.
    • (Major) No ablation studies: The different framework parts are presented as a group, and their necessity is never established, neither qualitatively, nor quantitatively. Again, the authors concede this in G.1, but it does mean that this should not have been present in the paper.
    • (Medium) Model agnosticity: The paper makes the claim that Colosseum is model-agnostic in multiple places, including the Introduction and Conclusion. However, the framework has only been evaluated on 2 models from the Gemini family, leaving this claim undersubstantiated. Section 5.2 also never specifies the harness advantage for Gemini-3.7-Flash. This is still, a relatively minor point compared to those above.
    • (Medium) Unclear task selection: The tasks in 5.1, while showing that progress has been made, do not establish how many were attempted, what the selection criteria beyond success were.
    • (Medium--Minor) TCSBench Bias: TCSBench uses Gemini-3.1-Flash as a correctness judge in its original implementation. Given everything evaluated here are stronger models from the same family, this carries risks of inaccurate judging. This can be further exacerbated when using a Gemini-3.7-Flash judge to select between proofs. There are also further concerns of contamination given the papers come from as far back as 2020.
  3. Little-to-no reproducibility -> The paper offers no supplementary material, no concrete instantiation of the framework, includes only abridged versions of the prompts, and no insight on how the open-problem tasks were selected.

  4. Point of Section 4 -> Section 4, in my reading, only acts as a qualitative justification for introducing overlap between aggregator groups. It is not clear whether any hyperparameters are quantitatively guided by this method, or what the indended reading is beyond the qualitative argument. Since the section takes up a prominent part of the paper, I request some additional clarification from the authors.

Overall, these points make this submission unfortunately seem like it was not well-designed to further research in the field, as it does not explain well what value it brings to the community, why it should be considered instead of alternatives, and does not produce sufficient implementation details for any of these points to be established by members of the community. The only positive that remains is hence the idea itself, whose value remains unsubstantiated.

Questions for authors
  1. Which of the experiments did you allow the harness to utilise web access? For TCS-Bench and Erdos it is explicitly stated as disabled, but no information is provided for the rest of the evaluation.

  2. How is the research memory instantiated? This is an important design choice, without specification on format, how retrieval is performed, and any limitations to said memory.

  3. When (which stage) is execution of code allowed in the framework in 5.3?

  4. Is there any termination condition outside of the global verifier accepting the proof? Otherwise, this harness has the potential to indefinitely utilise compute.

  5. Please address the concerns listed in the Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.