acceptodds
Under review as a conference paper at ICLR 2027

Knowing When to Stop: IGP-Bench and the Multi-Agent System GALOIS for the Inverse Galois Problem

Abstract

Frontier agents can now sustain hours of autonomous work. Greater capability may bring greater progress, but it may also carry an agent farther along a wrong route. Existing long-horizon mathematics benchmarks are hard to build, saturate quickly, or grade partial progress only with a language model, which makes it difficult to reliably measure how much effort an agent spent on each route it tried. We introduce IGP-Bench, a benchmark of long-horizon algebraic geometry problems in inverse Galois theory. Its first release has 200 problems in three tiers: 39 with published explicit realizations, 55 whose realizations are known to exist but have not been made explicit, and 106 that remain open. On every problem, a successful route consists of objects that are hard to find but easy to verify, and GAP and Magma check each of them deterministically. Agents record their trajectory as an artifact graph, from which we score verified progress, format compliance, tool use, and knowing-when-to-stop metrics: a useful-resource fraction and a stopping-efficiency score comparing how far an agent goes down incorrect routes with how far it goes down correct ones. We test eight frontier models over 80 runs on the five Mathieu-group tasks, each run with a 10-hour wall-clock limit and 500,000 output tokens. Together these tasks offer 32,979 candidate routes, of which only 13 lead to a realization. Surprisingly, we find that Claude Opus 5 has the highest mean progress (76.2), followed by GPT-6 Sol (62.1) and GPT-6 Astra (59.4), and that the gap between them is not one of construction ability but of identifying the right route and of where the budget goes. These gaps motivate GALOIS, an open-source multi-agent system with a lead agent, a strategy reviewer, and specialist sub-agents for its 36 written skills, in which a dedicated route rejection agent screens candidate routes for known obstructions in the background, so wrong routes are ruled out quickly. In reference runs for IGP-Bench, GALOIS with a Claude Opus 5.5 lead produces Magma-verified polynomials for all five Mathieu tasks in 10 to 126 minutes, including and , for which no single agent produced a polynomial. Tasks, verifiers, evaluation code, and GALOIS will be publicly released.

Primary area:
datasets and benchmarks
Keywords:
mathematical reasoning, benchmark, long-horizon agents, multi-agent systems, inverse Galois problem, verifiable evaluation, LLM agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ptLaOpenReview ↗
Rating
2
Confidence
1(−1)
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether AI agents can make progress on difficult mathematics problems while knowing when to abandon an unpromising approach. IGP-Bench contains 200 inverse Galois problems: each asks the agent to construct a polynomial with a specified Galois group. GAP and Magma check intermediate results as well as the final construction. The agent records which research route each step belongs to and which earlier results it uses. The benchmark then scores both mathematical progress and how the agent spends its resources. The main comparison covers eight systems on five Mathieu-group tasks, with 80 runs in total. The paper also introduces GALOIS, a system with a lead agent, specialists following written skills, and mechanisms for rejecting routes. Its separate reference runs produce verified polynomials for all five tasks, under different conditions from the main comparison (Sections 3–6).

Critical strengths
  • The benchmark gives useful credit for work that falls short of a final answer. IGP-Bench checks intermediate mathematical results with GAP and Magma and separates research points, a route milestone, and continuation credit. A verified final polynomial receives full continuation credit regardless of the route used. This makes the evaluation more informative than a success-or-failure score, and mathematical progress does not depend on an LLM judge's opinion (Sections 3.3–3.4).
Critical weaknesses
  • Stopping efficiency V_r measures accumulated route costs, without separating necessary work to discover an obstruction from unnecessary work after it is known (Section 3.4). Unknown routes also count as non-positive, including four runs that produced verified polynomials on unlisted routes, although final construction credit is preserved (Section 7). These scoring choices limit its interpretation as stopping quality; they are separate from GALOIS's distinction between verified rejection and deferral.
  • The evaluation's novelty needs clearer positioning. Agentic Abstention already evaluates timely versus delayed stopping (Luo et al., 2026, Section 4.2), which Section 2 does not adequately discuss. IGP-Bench adds costs and verified progress for individual research routes, but the scoring issues above weaken this contribution.
  • GALOIS sensibly distinguishes rejection based on verified mathematical evidence from deferral with unknown status. However, its reference runs also change the lead model, supplied skills, budget, and scoring procedure, so completing all five tasks does not isolate the benefit of route rejection (Section 4). A controlled ablation is needed.
Questions for authors
  • Could you separate the cost of discovering an obstruction from additional work after the evidence is available, and assess unknown routes separately from verified dead ends?
  • What does the route-level evaluation establish beyond the stopping-time measures in Agentic Abstention?
  • How do the results change after fixing the polynomial-normalization and route-recording errors in Appendix I.1?
  • Could you ablate GALOIS's route rejection with the model, skills, budget, and evaluator held fixed, reporting verified progress and total cost?
1 comment
AuthorsOpenReview ↗

Clarifications on Experimental Design, Related Work, Error Corrections, and Additional Ablation Experiments

Thank you for your comments to help us improve. We have added experiments and would like to clarify the interpretation and contribution of our evaluation.

1. Interpreting the route-level metrics.

This is a good question. We agree that the cost of discovering an obstruction should be distinguished from work performed after the evidence is available, while VrV_r measures aggregate resource allocation. We have specified explicitly the resource allocation at different stages and at different routes in our Appendix. The Astra trajectory in the appendix shows the agent continuing along a route even after computing sufficient evidence to reject it over Q\mathbb{Q}. This helps explain Astra's weaker stopping performance compared with Opus.

We also agree that unknown routes should be distinguished from verified dead ends. Empirically, however, most routes explored on the five Mathieu-group tasks did not lead to a verified realization within the stated budgets. This motivates our current labeling as an empirical approximation of route utility, while leaving the mathematical status of unresolved routes open.

2. What IGP-Bench adds beyond Agentic Abstention.

Agentic Abstention is relevant work on timely stopping. Our focus is on decisions among individual research routes within an ongoing research-level task that requires heavy computations and long-horizon reasoning.

A distinctive feature of IGP is its well-defined routes, such as (2A,23A,23B)(2A,23A,23B) for M23M_{23}, together with deterministic checks of intermediate results and obstruction certificates. Such a combination is difficult to obtain in frontier mathematics.

These routes involve large-scale computation and abstract reasoning. Beyond initial screening, agents must compute additional properties, interpret the results, and decide whether to continue, investigate an obstruction, or change strategy. This allows us to study mathematical judgment and research decisions throughout a long reasoning process. We believe that long-horizon research, clear evaluation criteria, and deterministic verification are the main value here, providing information that complements stopping-time measures.

3. Additional ablation experiments.

We added ten runs with the route-rejection and candidate-review advisory agents disabled. Across the five Mathieu-group tasks, the full and ablated configurations reached the specified endpoint in 10/10 and 9/10 runs, respectively. Costs measured up to the endpoint or termination were 265 versus 424 steps, approximately 930K versus 1,270K output tokens, and USD 144.65 versus USD 200.52 in API expenditure.

Reviewer_dj3kOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents 200 math problems as benchmark for the research agents. Compared with classical algebra systems and various LLMs being tested.

Critical strengths

I think it is quite reasonable and promising direction - to use math problems and tests for the research agents. The paper proposes quite carefully organised study in that direction.

Critical weaknesses

None

Questions for authors

In view of impressive progress of openAI internal model reported to solve hundreds important math conjectures - is is expected that this challenge will be saturated as soon as the model would be available ?

1 comment
AuthorsOpenReview ↗

New insights from IGP problem, Saturation, and added experiments on open cases.

Thank you very much for your positive comments and for raising the concern about possible saturation. We have added experiments to address this concern. Although we cannot predict the performance of an unreleased model, our results show substantial remaining difficulty for the evaluated systems under the stated budgets.

1. The experiments show that IGP-Bench remains challenging.

Our first release contains 200 problems, including 106 classified as open in the task catalog. Even on the five Mathieu-group tasks, which have published realizations, no model in our main single-agent evaluation produced more than 2 cases out of 5 cases, under the 10-hour and 500,000-output-token budget.

Beyond the Mathieu groups, we evaluated Opus 5, the best-performing system in our original study, on ten groups in an exploratory experiment. None produced a cover. We now also added a comparison of Opus 5.5 and Astra on five other new groups , with two runs per model per group. Under a 10-hour and 500,000-output-token budget per run, neither model obtained a verified final realization in any of these 20 runs. These results provide direct evidence of substantial remaining room for improvement across different problem families.

2. Intermediate results reveal both mathematical progress and difficulties in route selection.

We added 20 runs of new experiments on frontier models, Opus 5.5 and Astra. They made substantial, verifiable progress even when they could not complete the final construction. This allows us to distinguish different levels of capability among runs with the same final outcome.

These IGP tasks reveal some interesting findings of current agent's shortage. For example, in an M23M_{23} run, Astra recorded a rationality obstruction to a route early in the run, but still subsequently spent approximately 90,000 artifact output tokens constructing a cover on that route. Such behavior is difficult to identify from a final success-or-failure score alone.

3. Future improvements can contribute to both evaluation and mathematics.

Our best advantage is that IGP is not a yes/no question but a testbed for carefully studying how agents think, spend budget, and make decisions in long-horizon reasoning and large-scale computation. If future models solve more of these problems, IGP-Bench can still compare the verified progress, route choices, and computational costs of their solutions.

Reviewer_NxSUOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces IGP-Bench, a benchmark of 200 inverse Galois problems with deterministically verifiable intermediate checkpoints, and proposes metrics for evaluating long-horizon agents' ability to allocate computational resources and abandon unproductive research directions. The authors evaluate eight frontier models primarily on five Mathieu-group problems and introduce GALOIS, a multi-agent system incorporating domain-specific mathematical tools, specialist agents, and explicit route-rejection mechanisms.

Critical strengths
  • The benchmark provides deterministic verification of intermediate mathematical progress, reducing reliance on LLM-based judges and allowing more fine-grained analysis of long-horizon agent trajectories.
  • The authors put substantial engineering effort into constructing the benchmark and GALOIS system, with detailed instrumentation for tracking resource allocation, tool usage, and agent behavior across different research directions.
Critical weaknesses
  • The problem setting is extremely niche, and the broader scientific contribution to AI is insufficiently justified. Although knowing when to abandon an unproductive reasoning direction is an important general problem, the proposed benchmark, evaluation criteria, and agent architecture are heavily tied to the specific mathematical structure of inverse Galois problems. It is difficult to identify what fundamental understanding of AI agents this work advances beyond demonstrating a specialized mathematical problem-solving pipeline.
  • The empirical evaluation is very narrow to support the paper's broader conclusions. Despite introducing 200 benchmark problems, the main comparative evaluation covers only 5 Mathieu-group tasks, with merely 2 runs per model per task. This substantially limits the diversity and reliability of the findings. The paper therefore provides insufficient evidence that its observations characterize long-horizon mathematical reasoning more broadly, let alone general agent behavior.
  • The GALOIS system has very limited methodological novelty, and the central design choices are not experimentally validated. The framework basically combines established multi-agent orchestration patterns with extensive domain-specific expert procedures, deterministic mathematical tools, and manually designed rejection rules. It is unclear whether this constitutes a generalizable algorithmic advance in AI. More importantly, there are no controlled ablations isolating the effects of route rejection, specialist agents, or other components.
Questions for authors
  • Beyond the specific mathematical constructions in inverse Galois theory, what generalizable scientific insights about long-horizon agent reasoning or stopping behavior does this work establish? How would these insights apply to other agentic tasks, and can authors validate across tasks?
  • Given that the benchmark contains 200 problems, why is the main evaluation restricted to only 5 closely related tasks? Do the authors have evidence that the reported findings remain consistent across different problem families or difficulty tiers?
  • What are the key methodological innovations of GALOIS beyond combining existing multi-agent orchestration and domain-specific mathematical tools? Can the authors provide ablations isolating the contribution of the route-rejection mechanism and specialist agents?
7 comments
AuthorsOpenReview ↗

Clarifying the Generalizable Scientific Insights from IGP, with Additional Ablation Studies and Expanded Experiments

Thank you for your comments and we have added experiments to address your concern. We would like to clarify the motivation for IGP-Bench, our added evaluation, and the role of domain-specific agents in GALOIS.

1. Why the inverse Galois problem provides a valuable testbed for agents.

The inverse Galois problem offers an infinite family of problems combining long-context reasoning, proof construction, and large-scale numerical computation. Most problems remain open, and even the five Mathieu groups required several decades of work and substantially different methods.

A useful feature is the availability of thousands of well-defined routes, such as (2A,23A,23B) for M23M_{23}. This creates concrete opportunities to evaluate whether agents recognize when to abandon an unproductive route. The problems are difficult, the contexts are long, and agents must integrate computational evidence before deciding what to do next. Studying these problems with clear evaluation criteria and verifiable intermediate results helps us understand how agents reason, allocate resources, and fall short beyond what large-scale yes/no evaluations reveal.

2. Broader evaluation and the diversity of the tasks.

We expanded the evaluation by testing Opus 5, the best-performing system in our original study, on ten cases beyond the Mathieu groups. We also added a five-group comparison of Astra and Opus 5.5 in the revision. Each problem involves multiple subproblems, from recognizing exact algebraic coefficients from numerical approximations to abstract reasoning. In fact, these 20 problems have 202 intermediate problems in total: 27 for five M groups, 50 for ten groups registered before and 125 more intermediate problems for the newly added 5 groups. The results show substantial remaining room for improvement under the stated budgets.

Although the five Mathieu groups share a family name, their realizations involve different methods and checkpoints. Tuple selection is a common starting point, but deciding whether to work directly with G, through Aut⁡(G)\operatorname{Aut}(G), an extension G.2G.2, or a larger group introduces distinct research challenges.

Evaluating open-ended conjectures often requires assessing long natural-language proofs, for which expert review is difficult to scale. Deterministically graded benchmarks face the challenge of maintaining research-level difficulty while expanding the task set. IGP-Bench combines difficult construction problems with computationally verifiable intermediate objects. Examining each stage reveals information about route selection, progress, and resource allocation that a final success-or-failure score or an LLM judge alone cannot capture.

3. The tasks test capabilities beyond mathematics.

These tasks require coding, faithful artifact recording, and planning over long contexts under substantial memory demands and complex computational constraints. Agents must write Magma scripts, interpret outputs, use them to support mathematical arguments, and decide what to compute next. They must also decide whether to search for an obstruction or continue a construction. These decisions are difficult to assess when evaluation criteria are unclear; IGP provides concrete computational evidence against which to examine them. This interaction between reasoning, coding, evidence management, and resource allocation makes the benchmark a valuable stress test for agents more broadly.

4. GALOIS and a possible direction for multi-agent research.

GALOIS explores building specialized agents on increasingly capable general-purpose harnesses to tackle domain-specific difficulties. As frontier systems improve their built-in planning and orchestration, domain expertise may become an increasingly important source of additional capability.

GALOIS supplements standard planning and review roles. But it also adds the domain-needed obstruction agents that actively investigate whether routes can be ruled out using mathematical invariants, structural properties, and numerical evidence. They pursue specific, testable obstructions, with exact verification required before a route is rejected. This could suggest a broader design principle as well: rejection agents can use concrete properties and verifiable signals from the problem, rather than relying only on general critiques or attempts to find errors.

We have added ablation experiments to assess the contribution of these advisory components on difficult cases. Across the five Mathieu-group tasks, the full and ablated GALOIS configurations reached the specified endpoint in 10/10 and 9/10 runs, respectively. The corresponding resource costs were 265 versus 424 steps, approximately 930K versus 1,270K output tokens, and 144.65 versus 200.52 dollars in API costs, measured up to the endpoint or run termination.

Thanks authors for the response! I appreciate the response but I still hold concerns on several points I originally have.

The authors mentioned "the inverse Galois problem offers an infinite family of problems," but this does not address that the problem itself is very niche, and why people would care about having this evaluation in addition to hundreds if not thousands of existing evaluations for agentic capabilities.

I think the property of having an infinite family of problem imo is a great thing, because instead of building an evaluation out of it, this can be served as an environment for agentic RL training, and showing that a model RL'd from vast amount of problems improves general agentic capabilities more effectively than problems from other domains would be a much stronger result than turning this into another agentic evaluation would carry.

I concur that "the tasks test capabilities beyond mathematics," but evaluation of all these atomic (and/or combinations of) capabilities exists; I don't see sufficient motivation why we need another one that probes similar capabilities. For this specific types of math problems it is a useful evaluation, but again this goes back to my original concern on being too niche to the broader AI community.

I would like to acknowledge that I do not have direct research experience on mathematics and may have undervalued the importance of the inverse Galois problem for the mathematics community broadly, but from my background with more direct experience with agentic capabilities and evaluations, this seems very niche and does not provide many new insights that can be broadly beneficial. In its current form, my evaluation is that this can be a better fit for workshops relating to agents and mathematical reasoning, but not as a conference paper. Thus I remain my original rating and lean reject.

AuthorsOpenReview ↗

What IGP-Bench Can Measure and Why

Thank you for suggesting that IGP-Bench could serve as a platform for agentic reinforcement learning. We share your enthusiasm and plan to explore this direction in future work. We would also welcome potential collaboration after the conference.

We would like to emphasize that a key contribution of IGP-Bench is evaluating whether agents know when to stop pursuing an unproductive research route. Unlike final-answer evaluations, our benchmark examines how agents interpret evidence, make decisions, and allocate resources throughout a long research process that requires substantial calculations (10 hours).

Our experiments reveal an important failure mode: agents sometimes continue spending substantial computational resources on a route even after their own computations have produced sufficient evidence to rule it out. Unlike existing work on agentic abstention (Luo et al., 2026), identifying when to stop in our setting can itself require difficult mathematical reasoning, extensive computation, and verification of intermediate results.

IGP-Bench makes these decisions objectively verifiable. By combining computational tools, abstract reasoning, structured artifact graphs, and deterministic checkpoints, we can evaluate not only whether agents solve a problem, but also whether they recognize and act on evidence during the process.

This is also why we prioritize carefully designed problems over simply increasing benchmark size. Our five Mathieu-group cases provide established mathematical results and verifiable intermediate steps, while 15 additional cases broaden the evaluation. Together, they offer a balance between research-level difficulty and rigorous, process-level evaluation.

We believe that understanding when and why agents should abandon a research direction is a valuable research question beyond mathematics, and IGP-Bench provides a concrete setting for studying it.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions: direct measurement of stopping, GALOIS ablation, and broader evaluation

We thank all reviewers for their feedback. We have uploaded a revised PDF and new supplementary data; table numbers below refer to the revised version.

Measuring stopping directly. The main change is a route-level analysis of what agents do after they have evidence against a route (Section 6, Appendix G.8). Because every output token is attached to an artifact and every artifact to a route, we can date work relative to the agent's own findings. On M23M_{23}, in 12 of 16 runs from seven systems, the agent itself records that a class vector is not Galois-stable, so that no cover with it exists over Q\mathbb{Q}, and then keeps researching that class vector. This accounts for 1.14M output tokens, 21% of these runs' output, while the correct route (2A,23A,23B)(2A,23A,23B) gets 0.2%. By the agents' own declared intents, most of this work (83%) goes into computing the cover over a number field; none of it targets a cover over Q\mathbb{Q} (Tables 18–19). Neither the progress score nor KrK_r reveals this cost, which is why we now report it separately, together with the token split by route status for all runs (Table 17).

GALOIS ablation. We re-ran GALOIS on the five Mathieu tasks with its two advisory agents (the route-rejection agent and the candidate reviewer) switched off (Table 2). The full system reaches the endpoint in 10/10 runs and the ablated one in 9/10; the ablated runs also need more steps (424 vs. 265), more output tokens (1.27M vs. 0.93M) and more API cost (USD 200.52 vs. 144.65). In the failed run, on M22M_{22}, GALOIS abandons the working route after two failed curve recoveries and spends the remaining time on 29 other class vectors.

Beyond the Mathieu groups. We added a comparison of Claude Opus 5.5 and GPT-6 Astra, two runs each, on five groups with no known realization: PSL3(8)\mathrm{PSL}_3(8), PSU3(9)\mathrm{PSU}_3(9), Sz(32)\mathrm{Sz}(32), 2G2(27){}^2G_2(27) and 3D4(3){}^3D_4(3) (Table 11). Neither system finds a polynomial, but they differ on the early stages: Opus 5.5 gets the full screening score on all five groups, Astra 83–95% on three of them.

Other changes. KrK_r is now described as a route-allocation metric and reported with a total-cost variant KrtotK_r^{\mathrm{tot}} (Table 8). A route now also counts as correct if the literature supports it for the automorphism group of the target, or if the run itself produces a verified polynomial on it (Section 3.2). This raises mean PP for four systems: Claude Opus 5 from 76.2 to 81.0, GPT-6 Astra from 59.4 to 64.0, GPT-5.6 Sol from 38.8 to 44.2, and Claude Sonnet 5 from 11.5 to 17.5. Table 3 also includes a new system, Claude Fable 5.1. The related-work section now discusses agentic abstention benchmarks (AgentAbstain; Luo et al., 2026), which ask whether an agent should give up a task rather than which routes it pursues within one. The supplementary data contain the per-run scores, the route-status split, all post-evidence artifacts with the agents' declared intents, and the GALOIS polynomials with their Magma checks.

We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.