- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a Planner-Executor-Feedback workflow for maths agent. Planner decompose the problem into a 10 steps plans and determine what specific maths atomic capability should be used in this steps; the skills are then loaded into executor to make progress; after each step, the verifier checks the observations and determine whether retry, continue or stop. The paper reports gains over base models on three text and three multimodal math benchmarks, outperforms AutoGen / OctoTools / MathAgent while being more token-efficiency.
Critical strengths
- Shows consistent improvement among a range of maths benchmark and across both open-source and closed-source models, and beats other agent frameworks with fewer tokens used. This is helpful because the comparison is not depend on the model backbone and tool use or not. There is evidence that the capability typed decomposition itself has its value.
- The atomic capabilities are curated and discussed by domain experts, and the ablation shows it matters: replacing it with generic labels costs +4.6 / +2.8 (Qwen3-8B) and +5.4 / +4.3 (Gemini-3-flash). Also these principles might be valuable beyond this agent framework.
Critical weaknesses
-
The paper does not specify what the validation contract actually is, although it is important to understand how verifier works based on the atomic capability. It is only described as three lists (operational / local / global checks) generated by the planner, with no explanation of what a check looks like or how it is evaluated. In Figure 9, the verifier prompt is: "Rules:\n- If the current step number is less than the predefined total number of steps and the previous step has a valid result, CONTINUE.\n- If the current step number is greater than or equal to the predefined total number of steps and a complete solution is available, STOP.\n- If a step fails or the result is empty, but there are still steps left, CONTINUE.\n- STOP only when all necessary steps have been completed and the final answer has been obtained.\n". There is no verifier contract here. This matters because the paper’s central claim is that gains come from capability-grounded planning and verification (Sec 3.5, ablation on replanning); if the verifier that actually ran is a generic step counter, that part of the claim is unsupported.
-
The reported AIME25 score for Gemini-3-Flash (60.0) is far below the score shown on MathArena (97.5%, with thinking_level=high, https://matharena.ai/models/gemini_gemini_3_flash). MathArena’s output is about 18K tokens per problem, which is within the paper’s 20K output limit. Since the main gains are measured against the base models, this needs explanation.
Questions for authors
- For each base model evaluation, please specify the configuration such as the thinking setting and output limit. Please explain the differences between your eval and public leaderboard scores.
- Please confirm how the Figure 4 (a) end to end token composition was calculated. For base model, if it is without any agent flow, how can it has 34K (input + output) per problem when your output token limit is 20K. (Does it mean in average your problem input is more than 14K?)
- Which verifier actually ran: the contract-aware one in B.2, or the step-counting one shown in Figure 9? Could you give me an example of the validation contract, or explain it better?
2 comments
Response to Reviewer iDkv (Part 2/2)
W2 & Q1: Gemini-3-Flash gets 60.0 on AIME25, far below MathArena. Please specify thinking settings and output limits for every base model.
Response
Thank you for pointing out that the original submission gave too little detail about the inference settings. To make the comparison within a backbone fair, we evaluate Base and AtomMath-Agent under the same model settings in Table 1. For Gemini-3-Flash, both use thinking_level=medium and a per-request output cap of 20,000 tokens. They also use the same problems, answer extraction, normalization, and scoring procedures.
The public MathArena entry lists thinking_level=high and a 64,000-token output cap. These settings differ from ours, so the public score and our result reflect different evaluation conditions. The average output length on the leaderboard also leaves open how many individual responses would reach the 20K cap. Other inference settings, API variation, evaluation dates, and truncation could affect the results. We have clarified the settings so that readers can interpret Table 1 as evidence of the relative benefit of our method under the stated, shared inference configuration. MathArena model page.
The revised text specifies the Gemini settings and the agent's request and round limits.
Revised manuscript. We supplement the Implementation and Evaluation Details in Page 20, Line 1026-1046.
Q2: How can Base LLM use about 34K tokens per problem with a 20K output limit?
Response
Thank you for pointing out the ambiguity in our budget description. For AtomMath-Agent, 20K is the output cap of each individual LLM request. The efficiency experiment uses a separate setting for the unwrapped Base model. To reduce truncation on individual problems and better assess the token–accuracy trade-off, we raised its per-request output cap to 42K. With a 20K cap in that experiment, responses that need more than 20K output tokens could be truncated, limiting the range of the efficiency comparison. Figure 4(a) reports an average of 8,453 input tokens and 25,465 output tokens, for 33,918 tokens per problem in the higher-cap efficiency setting. The Table 1 Gemini Base uses the 20K setting described above.
AtomMath-Agent can execute up to 10 solve rounds. A round can include multiple LLM requests for planning, execution, verification, and retry handling. When the problem is solved or the maximum number of rounds is reached, the system generates a final summary and returns its answer. We calculate end-to-end token use by summing the inputs and outputs of all requests for each problem, including retries within a round and the final summary. We have added these settings and the accounting procedure to the revised manuscript.
Revised manuscript. We supllement the settings in Page 20, Line 1026-1057.
Thank you again for the detailed feedback. We hope the contract example and the clarified evaluation settings address the concerns about the verifier and the comparisons. The relevant changes are highlighted in the revised manuscript, and we will incorporate them into the camera-ready version.
Response to Reviewer iDkv (Part 1/2)
Thank you for your careful reading of the paper and for pointing out where the verifier description, inference settings, and token accounting needed more detail. We hope the following clarifications make the experimental setup and the role of the verifier easier to understand.
Response
Thank you for pointing out how Figure 9 could be read. Our experiments used the contract-aware verifier, with different validation conditions for different atomic capabilities. Figure 9 presents an abbreviated control-flow prompt and omits the current atomic unit, the capability-specific contract, and the full execution evidence. We agree that the figure did not give enough detail to show how the verifier reached its judgments. We have clarified the inputs and decision rules used in the experiments and aligned the explanations in the main text, appendix, and figure caption. The step count tracks whether the solve loop has reached its round limit.
In our interface, the validation contract is passed to the verifier with the atomic unit. The unit's mathematical capability determines the kinds of conditions to check. The subgoal and verified dependencies turn those conditions into requirements for the particular step. Operational checks assess whether execution succeeded. Local checks assess whether the output meets the current mathematical task's requirements. Global checks assess whether the current results are sufficient to answer the original problem. A successful execution can still fail a mathematical check. The round limit ends the loop when the budget is reached, and the verifier judges completion from the contract and available evidence.
Consider a Calculation unit for the following problem. Find all real solutions of x+2=x and return their sum plus one.
solutionsfield, with a derivation or calculation evidence for the candidates.solutionsfield is present and parseable, and the record indicates no execution error or truncation.Suppose the Executor returns {−1,2} after squaring the equation. The operational check passes, but −1 violates L1 and fails L2 because substituting it gives 1=1=−1. The verifier can identify the issue as an extraneous root introduced by squaring and return Retry, with the failure evidence for −1. The Planner can then revise or retry the unit, retaining the valid solution set {2}.
Once {2} passes the local checks, the overall task still requires one more step. The verifier returns Continue and releases the successor unit that depends on the solution set. The successor computes 2+1=3. The verifier returns Stop once all parts of the original objective are supported. The same contract thus guides candidate checking, local repair, and the use of the result by later units.
The LLM verifier receives the substitution, constraint, and completeness requirements as written conditions, then judges the execution result against them. For a Theorem Application unit, the contract would instead focus on the theorem's premises and scope of applicability. We hope this example makes it clearer how the check conditions follow from the mathematical capability assigned to the unit.
Revised manuscript. Validation, Feedback, and Memory in Page 6, Line 307-310, verifier inputs and workflow caption Page 15, Line 776-780, and the worked Calculation contract in Page 15, Line 806-809 and Page 17, Line 864-871.