- Rating
- 2
- Confidence
- 1(−1)
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether AI agents can make progress on difficult mathematics problems while knowing when to abandon an unpromising approach. IGP-Bench contains 200 inverse Galois problems: each asks the agent to construct a polynomial with a specified Galois group. GAP and Magma check intermediate results as well as the final construction. The agent records which research route each step belongs to and which earlier results it uses. The benchmark then scores both mathematical progress and how the agent spends its resources. The main comparison covers eight systems on five Mathieu-group tasks, with 80 runs in total. The paper also introduces GALOIS, a system with a lead agent, specialists following written skills, and mechanisms for rejecting routes. Its separate reference runs produce verified polynomials for all five tasks, under different conditions from the main comparison (Sections 3–6).
Critical strengths
- The benchmark gives useful credit for work that falls short of a final answer. IGP-Bench checks intermediate mathematical results with GAP and Magma and separates research points, a route milestone, and continuation credit. A verified final polynomial receives full continuation credit regardless of the route used. This makes the evaluation more informative than a success-or-failure score, and mathematical progress does not depend on an LLM judge's opinion (Sections 3.3–3.4).
Critical weaknesses
- Stopping efficiency V_r measures accumulated route costs, without separating necessary work to discover an obstruction from unnecessary work after it is known (Section 3.4). Unknown routes also count as non-positive, including four runs that produced verified polynomials on unlisted routes, although final construction credit is preserved (Section 7). These scoring choices limit its interpretation as stopping quality; they are separate from GALOIS's distinction between verified rejection and deferral.
- The evaluation's novelty needs clearer positioning. Agentic Abstention already evaluates timely versus delayed stopping (Luo et al., 2026, Section 4.2), which Section 2 does not adequately discuss. IGP-Bench adds costs and verified progress for individual research routes, but the scoring issues above weaken this contribution.
- GALOIS sensibly distinguishes rejection based on verified mathematical evidence from deferral with unknown status. However, its reference runs also change the lead model, supplied skills, budget, and scoring procedure, so completing all five tasks does not isolate the benefit of route rejection (Section 4). A controlled ablation is needed.
Questions for authors
- Could you separate the cost of discovering an obstruction from additional work after the evidence is available, and assess unknown routes separately from verified dead ends?
- What does the route-level evaluation establish beyond the stopping-time measures in Agentic Abstention?
- How do the results change after fixing the polynomial-normalization and route-recording errors in Appendix I.1?
- Could you ablate GALOIS's route rejection with the model, skills, budget, and evaluator held fixed, reporting verified progress and total cost?
Clarifications on Experimental Design, Related Work, Error Corrections, and Additional Ablation Experiments
Thank you for your comments to help us improve. We have added experiments and would like to clarify the interpretation and contribution of our evaluation.
1. Interpreting the route-level metrics.
This is a good question. We agree that the cost of discovering an obstruction should be distinguished from work performed after the evidence is available, while Vr measures aggregate resource allocation. We have specified explicitly the resource allocation at different stages and at different routes in our Appendix. The Astra trajectory in the appendix shows the agent continuing along a route even after computing sufficient evidence to reject it over Q. This helps explain Astra's weaker stopping performance compared with Opus.
We also agree that unknown routes should be distinguished from verified dead ends. Empirically, however, most routes explored on the five Mathieu-group tasks did not lead to a verified realization within the stated budgets. This motivates our current labeling as an empirical approximation of route utility, while leaving the mathematical status of unresolved routes open.
2. What IGP-Bench adds beyond Agentic Abstention.
Agentic Abstention is relevant work on timely stopping. Our focus is on decisions among individual research routes within an ongoing research-level task that requires heavy computations and long-horizon reasoning.
A distinctive feature of IGP is its well-defined routes, such as (2A,23A,23B) for M23, together with deterministic checks of intermediate results and obstruction certificates. Such a combination is difficult to obtain in frontier mathematics.
These routes involve large-scale computation and abstract reasoning. Beyond initial screening, agents must compute additional properties, interpret the results, and decide whether to continue, investigate an obstruction, or change strategy. This allows us to study mathematical judgment and research decisions throughout a long reasoning process. We believe that long-horizon research, clear evaluation criteria, and deterministic verification are the main value here, providing information that complements stopping-time measures.
3. Additional ablation experiments.
We added ten runs with the route-rejection and candidate-review advisory agents disabled. Across the five Mathieu-group tasks, the full and ablated configurations reached the specified endpoint in 10/10 and 9/10 runs, respectively. Costs measured up to the endpoint or termination were 265 versus 424 steps, approximately 930K versus 1,270K output tokens, and USD 144.65 versus USD 200.52 in API expenditure.