- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces SAT, which is a fixed three LLMs team whose way of collaborating is learned by reflective evolutionary search on a small training set. On five math and physics benchmarks, SAT averages 66.2%, which beats the best member, a compute-matched single-agent control, and a routing oracle that perfectly selects among the member's independent answers. The paper also argues that demonstrability (whether correct reasoning can be told apart from incorrect ones) explains when collaboration helps.
Critical strengths
This paper is well-written. The question is important and motivated. The routing oracle is a valuable contribution. Details are pretty transparent. The full strategy banks are released, and reporting coverage alongside accuracy cleanly separates generating correct solutions from selecting them.
Critical weaknesses
-
The oracle picks from a single answer per member, while SAT picks from 10 multi-agent conversations, so the comparison mixes up interaction with sample count. I think a better reference should be oracle coverage over many independent samples per member at matched token cost.
-
Although the authors implemented a wide range of baselines already, I find the effect of learning is still confounded with pure ensembling and selection. Particularly, SAT runs 10 strategies and use an o3-mini auditor to choose among them. The initial strategy is run once with majority vote, and no baseline gets a comparable judge. The gains thus cannot be credited to learned organization until the same judge is applied to equally sized baseline pools. Could you also report each strategy's accuracy on its own?
-
Gains over the strongest control seem small and statistical uncertainty is not reported. SAT beats linearization by 3.7 points on AIME26 and ties it on HMMT26. On the knowledge suite the margin is 0.7. In your experiments, the in-distribution AIME24 test set has only 15 problems and there are no CIs or paired tests.
-
I think the framing somehow overstates. There is one member designing the strategies offline, and the bank is a frozen one, and a central judge picks the answer. This to me is a learned protocol ensemble rather than true self-organization as described in the text.
Questions for authors
- Figure 7 mentions earlier searches: how many search runs and team rosters were tried, and how was the final setup chosen?
- What are the per-problem token costs for each method, and what does each strategy in the bank achieve alone?
- Do the results hold with current frontier models? Could Claude Sonnet 4 have seen AIME 2025 problems during training?
Hi everyone, please acknowledge the authors’ rebuttal with a brief reply, even if your assessment is unchanged (if you already did or are about to do so, please ignore this message).