- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a new agentic framework for research-level math problem-solving, called RMA. This workflow addresses the inconsistency issue in long-horizon math reasoning with a typed research store and a context orchestrator. In the following experiments with human-expert evaluation, this workflow constantly exhibits superior performance over several baselines.
Critical strengths
- The superior performance of RMA is supported by many experiments on frontier math research. The authors also reported the cost and scalability of each experiments, providing more information on the efficiency of RMA.
- The inconsistency of agents in the long-horizon math research is an important issue in related research. This work is actually trying to address some realistic and meaningful drawbacks.
- The authors have done substantial experiments on research-level math problems. The human-expert evaluation do require a large amount of work.
Critical weaknesses
- The generalizability might be limited by the complex workflow design of RMA. Its implications for subsequent work would therefore also be limited.
- Some experiments and claims have minor reliability issues (See Question 2 below).
Questions for authors
- Among all parts in the agent workflow design, which one do you believe has the highest contribution to the final performance?
- The performance in the component ablation study (Figure 2, c) seems too good to be true. I am wondering about the randomness in these experiments. Since These experiments used a complex workflow with LLM judge and the dataset only consists 30 problems, I believe there should be notable variance in the score estimation. How do you get a monotone performance gain in the ablation study? The claim that all components be effective is not very convincing.
- Have you considered comparing with some general-purpose agents such as OpenAI Codex or Claude Code on these long-horizon math problems? These are, as far as I know, famous tools that a mathematician will use in their daily work as well. So I believe this comparison would be very informative.
Minor Issues or typos:
- The right subfigure of Table 2 is not correctly sorted by pass rate as declared in the table.
4 comments
Response to Reviewer Tu3n (part 2/2)
Q1. Which part contributes most to final performance?
Persistent state and bounded context construction produce the largest degradations when weakened. In the plotted ablation, replacing the store with a stateless variant reduces the score from 8.5 to 2.6. Replacing bounded context construction with context dumping reduces it to 3.4. The repeated means are 2.63 ± 0.25 and 3.47 ± 0.21, compared with 8.53 ± 0.15 for full RMA. The more controlled summary-memory comparison strengthens the conclusion about persistent state. It retains the full workflow and yields 6.63 ± 0.15, with 16/30 human passes compared with 25/30 for full RMA. The context-dumping result requires a narrower interpretation. Dumping also produces oversized or failed calls and fewer completed operations. Its degradation therefore reflects a joint information-and-resource effect, rather than a pure effect of irrelevant context. These results place persistent research state and controlled exposure of that state among the most consequential mechanisms in the tested architecture.
Q2. How are the ablations scored, and why do the results look consistently ordered?
The continuous score measures proof correctness and completeness. Candidate proofs are anonymized and scored from 0 to 10 by Opus 4.8, GPT-5.2, and Gemini 3.1 Pro. Their scores are averaged for each proof and then across problems. Each proof is assessed once by each judge by default. Repeated experimental results come from repeated system runs, rather than repeatedly judging one fixed proof. Same set of problems are used in all ablation studies.
Figure 2(d) contains three separate comparisons:
- Stateless memory, previous-round-only memory, and full persistence.
- Context dumping, naive truncation, and operation-specific compilation.
- FIFO, severity-only, and severity-plus-dependency issue ordering.
The intermediate settings are included to test milder alternatives, rather than relying only on extreme component removal. Some intermediate settings are single-run measurements, so their exact ordering should not be interpreted as a statistical ranking.
Blinded mathematician evaluation gives 25/30 passes for full RMA and 16/30 for summary memory. On 13 problems, only RMA passes; on 4, only summary memory passes (two-sided exact McNemar p = 0.049).
The result order question is answered in W2.
Q3. Comparison with Codex or Claude Code.
Most of our API budget is allocated to Claude models, so our controlled workflow comparisons primarily use Claude backbones. We use GPT-5.2 and Gemini 3.1 Pro alongside Opus 4.8 for evaluation under a small budget.
We compare different agent frameworks under the same backbone across several experiments: RMA and GIR use Opus 4.8, identical tools, and $185 per problem on First Proof B2; RMA and agentic baselines use Opus 4.8, identical tools, and same budget per problem on held-out ResearchMath. The summary-memory comparison also retains RMA's backbone and workflow while changing state representation and retrieval.
We have additionally included a three-backbone comparison in Section 4.2. For Opus 4.6, Opus 4.8, and Fable 5.1, we compare RMA with GIR equipped with RMA's output-selection rule on the same 30 held-out problems, using identical tools and the cost cap. RMA leads by 2.32, 3.03, and 3.08 points, respectively. These comparisons prioritize identifying the effect of workflow organization under shared conditions; a direct Codex or Claude Code comparison would additionally evaluate those systems' own orchestration and context management.
Minor comment. Sorting in Table 2.
Thank you for pointing this out. We have reordered the Formal Conjectures panel in Table 2 by verified count, from highest to lowest.
Overall, we have clarified that Figure 2(c) reports separate component ablations sorted by score, provided repeated-run results, and added a three-backbone comparison to Section 4.2. The repeated experiments and blinded human evaluation support the value of structured research state and bounded context construction. We also expanded Related Work and the conclusion to explain the design's relevance beyond mathematics, and corrected the ordering in Table 2. If these results and clarifications address your concerns, we would greatly appreciate it if you would consider raising your score.
References:
[1] Rajasekaran, P. (2026). Harness design for long-running application development. Anthropic, March 24. https://www.anthropic.com/research/harness-design-long-running-apps
[2] Woo, Y., Lee, W., and Huang, J.-X. (2026). YouRA: A Persistent-State Architecture for Evidence-Traceable Autonomous Research Agents. arXiv:2610.01097. https://arxiv.org/abs/2610.01097
I have read your rebuttal and I believe my major concern in soundness is resolved. The deviation of scores in the repeated experiments reported is very informative. I think the contribution of this paper to the ML or AI4Math community is still incremental. Given the limited significance I will not raise my score to 4. Nevertheless, I would like to see this paper being accepted for their hard work and concrete experimental results.
We sincerely thank the reviewer for their thoughtful feedback and for acknowledging that our rebuttal has resolved the major soundness concern. We are also glad that the additional repeated experiments and reported score variations were informative.
Regarding significance, we believe our work provides valuable insights into the practical challenges and reliability of AI for mathematical reasoning, supported by systematic experiments and concrete empirical evidence. We thank the reviewer again for their constructive feedback and encouraging comments.
Response to Reviewer Tu3n (part 1/2)
We thank you for recognizing the importance of long-horizon consistency, the breadth of the research-level experiments, and the effort involved in expert evaluation.
W1. Generalizability of the workflow.
RMA targets mathematical research, while its implications extend to how agents organize state and coordinate extended work.
Mathematical research is itself a substantial and demanding application domain. Developing research-level proofs requires tracking assumptions, unresolved lemmas, failed attempts, and dependencies across an evolving argument. RMA's operations address these requirements. Our evaluation spans informal proof development and Lean-verified theorem proving, with mathematician assessment and kernel verification. Improving agents' ability to develop research-level mathematics is a meaningful objective in its own right.
Recent work shows that related design needs extend beyond mathematics. Anthropic's 2026 study of long-running application development [1] coordinates planner, generator, and evaluator agents through structured files and explicit acceptance criteria. YouRA [2], released in October 2026, maintains hypotheses, execution evidence, and failure histories to control recovery and review in autonomous research. These systems illustrate a broader challenge: agents must preserve unfinished work and accumulated evidence while deciding what to do next. RMA examines this challenge through proof-specific state and issue-directed orchestration.
Subsequent systems can adapt the state–operation interface to their own tasks. In software engineering, unresolved proof obligations could become failing tests or unmet requirements, with attempted patches and test results retained for later operations. In experimental research, the records could represent hypotheses, unsuccessful experiments, and outstanding validation checks. The operations and validation tools would change, while the store and orchestrator could preserve dependencies, retrieve relevant prior work, and record outcomes. Our ablations provide concrete evidence about the effects of persistent state, context construction, and scheduling that can inform these adaptations.
We have revised the related works section to reflect this discussion and we have revised the conclusion section to discuss future directions.
W2. Randomness and the ablation results.
The monotone appearance comes from sorting the measured scores for display. Each component is tested separately against the full system; the bars do not represent successive component removals or additions. We have clarified this in the Figure 2(c) caption and the “Component ablations” paragraph in Section 4.2.
Each bar is a separate experiment. For example, the literature ablation retains meetings and coordinated revision, while the meetings-and-revision ablation retains literature grounding. We evaluate these configurations individually and then arrange their scores in ascending order. The visual ordering therefore does not imply a monotone sequence of experimental improvements.
We quantify run-to-run variation through repeated experiments, summarized in the table below. Six central configurations are evaluated over three runs on the same 30 problems. Full RMA scores 8.53 ± 0.15, severity-only ordering 7.23 ± 0.15, context dumping 3.47 ± 0.21, and the stateless store 2.63 ± 0.25. Here, ± denotes the standard deviation across run-level mean scores. Each proof is anonymized and assessed by the same three-model judge panel.
The repeated results support the larger observed effects. Sorting the single-run bars does not establish significance between nearby configurations. The central state-management finding also receives separate blinded mathematician evaluation: full RMA achieves 25/30 passes versus 16/30 for summary memory, with a two-sided exact McNemar p = 0.049.
Table. Repeated key architectural comparisons