- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Stellar Colosseum is a many-agent workflow for research in mathematics and theoretical computer science. It explores strategies before building a proof, uses a readiness gate before decomposition, and repairs sections using their dependency graph. Within each stage, overlapping groups of candidates and critiques are synthesized into a final output. The paper also analyzes evidence retention in an idealized aggregation DAG. Evaluation includes five research contributions, two case studies, TCS-Bench, and Codeforces. It reports 71.0% on TCS-Bench with cross-model selection and 218/222 on Codeforces with execution feedback.
Critical strengths
- Keeping critiques attached to candidates is a useful design choice. A concrete objection can survive aggregation even when most candidates miss the same gap. The dependency graph and local repair also suit long proofs. Theorem 1 gives a clear analysis of evidence retention under explicit assumptions, including the logarithmic regime at the critical threshold.
- On TCS-Bench, Gemini 3.1 Pro improves from 30.3% to 54.0% with the harness (Table 3). The Codeforces evaluation checks accepted programs against the complete hidden tests using the original checkers. I also give weight to the coreset improvement and the long proof drafts for Knuth's cycles. These provide evidence from research tasks beyond the benchmarks.
Critical weaknesses
- It is hard to separate the effect of the design from the larger inference budget. Table 1 gives tree widths, but not actual token use, runtime, or retry cost. The 71.0% result requires two full harness runs and eight additional critiques. There is no baseline using repeated sampling and verification at the same budget, nor an ablation of overlap or critique retention. The Codeforces comparison isolates execution feedback only. Appendix G.1 acknowledges these gaps. The retention theorem studies evidence survival under independence assumptions; it does not establish which components improve proof success.
- I would like more detail on validation and human involvement. Section 5.1 states that the authors of the cited papers checked the mathematics, but neither that section nor Appendix D gives a run-level record of interventions or corrections. The unit-distance example is presented as strategy rediscovery; the verification status of its 22-page draft remains unclear. TCS-Bench relies on an automated grader. Its reported accuracy above 90% on 100 expert-labeled proofs does not establish its false-acceptance rate on the harness outputs. The AUC of 0.896 also uses this grader's labels.
Questions for authors
- What were the token use and wall time per problem for the Gemini 3.1 Pro TCS-Bench run, including verification and retries? A useful additional experiment would compare with repeated sampling and verification at the same budget. Could you also test one ablation of overlap or critique retention? A representative subset would help.
- Was the grader's reported accuracy above 90% measured on held-out examples or on the 100 proofs used to tune its prompt? Has an expert audit checked final harness proofs, especially cases where the internal verifier and cross-model selector disagree? If so, please report the audit size and the numbers of false acceptances and false rejections.
- For one result in Table 2, could you share the initial task, unedited harness output, and a short log of human interventions and later corrections? For the unit-distance draft, which claims have been independently checked and which remain unverified?