acceptodds
Under review as a conference paper at ICLR 2027

Agent-Guided Process Verification for Mathematical Self-Correction

Abstract

Large language models have made substantial progress in mathematical reasoning, yet in complex multi-step reasoning, an error at any step may propagate along an incorrect path, and the resulting solution may omit necessary branches. Existing natural-language reflection methods often struggle to locate critical errors precisely, while outcome verification typically checks only whether the available candidate answers satisfy the problem conditions and can therefore overlook necessary branches. To address these limitations, we propose Agent-Guided Process Verification (AGPV) for mathematical self-correction. AGPV decomposes the current solution into an ordered sequence of key claims, verifies them individually while invoking a shared mathematical toolbox when needed, and stops upon detecting the first error. A summarization agent converts the failure evidence into a diagnosis that guides a correction agent to re-derive the subsequent solution from that point; AGPV then adapts the next verification iteration to the revised solution. AGPV also verifies solution completeness. We further prove that AGPV guarantees solution correctness and completeness when key claims provide sufficient coverage and local verification is reliable, provided that all key claims pass verification. The implementation compresses long working contexts and reuses successful tool results within each problem. We evaluate AGPV on five challenging mathematical reasoning datasets with four models from DeepSeek, Qwen, and GPT. AGPV achieves an average accuracy of 63.8%, outperforming the strongest among the representative baselines evaluated (50.3%) by 13.5 percentage points, with higher repair coverage than the programmatic-verification baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, mathematical reasoning, process verification, self-correction, multi-agent systems

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_maWxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Agent-Guided Process Verification (AGPV) for mathematical self-correction. With multiple agents, AGPV detects and correct the first error in the key claims. The experiments prove its effectiveness on multiple math benchmarks.

Critical strengths

The proposed method is explained clearly.

The experiment is comprehensive. By comparing with multiple baseline methods on multiple benchmarks, the effectiveness of the proposed method is proven.

Critical weaknesses
  1. This framework uses multiple agents, which is likely to cost a lot more tokens. It is better to comprehensively compare the cost across all the methods. In Line472, the author claims that “Full per-call usage is unavailable across AGPV, ProgCo-Py, and Self-Refine”. Why not available?

  2. The prompt for each agent is missing in the paper. It would help understand the details if the prompt is written in the paper.

Questions for authors

In Table 1, Qwen3.6 35B-A3B-FP8 performs better than other models using Initial method, which is surprising and counterintuitive as DeepSeek V4.1-Flash should be much better than Qwen3.6 35B-A3B-FP8. How to explain this?

1 comment
AuthorsOpenReview ↗

Response to Reviewer maWx

Thank you for your suggestions. We have added a comparison of computational costs across methods and initial-answer results for the two models. We address each question below.

Q1. Computational costs by method

We measured the mean cost per problem for each method using DeepSeek Flash on five datasets.

MethodAccuracyModel calls (count)Total tokens (thousands)Python runs (count)Time (min)
Shared initial answer52.9%1.866.5100.57
Reflex60.0%3.0019.3700.61
Self-Refine61.8%6.0035.9500.92
Self-Reflection55.3%3.0022.2100.77
CheckList55.3%4.0019.9600.65
CoT-Tool54.1%3.0010.8700.72
ProgCo58.2%7.8257.2401.07
ProgCo-Py62.4%48.5517,004.4939.7931.76
AGPV72.4%14.7189.234.062.07

(1) Compared with standard feedback-based correction methods, AGPV used more tokens and achieved higher accuracy. For example, AGPV outperformed Self-Refine by 10.6 percentage points. The additional input was mainly used for stepwise verification: the model reads the reasoning steps and tool outputs before revising the solution.

(2) Compared with ProgCo-Py, which also uses Python, AGPV achieved 10.0 percentage points higher accuracy with nearly 90% fewer Python executions per problem. AGPV moves to correction after the first failed step and skips the remaining checks. When the same tool request recurs within a problem, AGPV reuses its previous successful result, avoiding 17 duplicate Python executions in this experiment. At the soft limit on consecutive tool calls, the system prompts the model to answer using the results already available. At the hard limit, it ends the current tool-call loop.

Q2. Prompts for the agents in Figure 2

The supplementary code submitted with the paper includes the prompts for all six agents in Figure 2. The text immediately before Table 3 in Appendix A of the revised manuscript gives the location of the full prompt file. The table below lists the core instructions for each agent.

Agent in Figure 2Core instruction (abridged)
Agen\mathcal A_{\mathrm{gen}} Generate key claims and a verification planExtract the key claims from the current solution and plan the checks in reasoning order. Include the problem's requirements for a complete answer.
Aver\mathcal A_{\mathrm{ver}} Verify key claimsCheck each claim according to the plan, using the mathematical toolbox as needed. Stop at the first failed claim and record the evidence.
Asum\mathcal A_{\mathrm{sum}} Summarize evidence of failureUse the verification record to identify the failed step, explain why it failed, and indicate where correction should begin.
Afix\mathcal A_{\mathrm{fix}} Revise the solutionUse the feedback to retain the valid reasoning, redo the derivation from the error onward, and provide a complete solution.
Aupd\mathcal A_{\mathrm{upd}} Update the key claims and verification planWhen the answer stays the same after correction, recheck the original plan against the revised solution and verification record. Retain applicable checks, add missing ones, and adjust those that no longer match the reasoning.
Atool\mathcal A_{\mathrm{tool}} Shared mathematical toolboxTurn mathematical requests from the other agents into executable Python code, perform symbolic solving, enumeration, or constraint checks, and return the results.

Q3. Initial-answer accuracy of Qwen and DeepSeek Flash

We compared the models' initial answers using the same test problems and scoring procedure. Overall accuracy is computed across all test problems, with equal weight per problem.

DatasetQwen initial accuracyDeepSeek Flash initial accuracy
AIME 202473.3%73.3%
AIME 202560.0%73.3%
BRUMO 202573.3%86.7%
HMMT 202556.7%53.3%
AMO Bench24.0%8.0%
Overall53.5%52.9%

Qwen and DeepSeek Flash each led on two datasets and tied on the fifth. Qwen's initial-answer accuracy on AMO Bench was 16.0 percentage points higher. In problem counts, Qwen's extra correct answers on AMO Bench exactly offset DeepSeek Flash's combined lead on its two stronger datasets. The overall gap comes from HMMT 2025, leaving Qwen ahead by 0.6 percentage points.

Reviewer_DRQHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed AGPA, a tool-augmented multi-agent framework for process correction of math reasoning. The authors used multiple benchmarks to validate their methods and show good performance against baselines. In addition, it shows that AGPV can improve more with more iteration.

Critical strengths
  1. The method decomposes the process verification as listing key claims, verifying key claims with tools, and process correction if errors are found. The workflow of such a decomposition makes sense.
  2. The authors use diverse math benchmarks to show the good performance of the proposed method
  3. The authors did some interesting analysis to further show the advantages of their method, such as better correction coverage
Critical weaknesses

Major issues:

  1. The authors do not validate the proposed method in terms of the motivation in the introduction. In the introduction, authors mention that drawbacks of existing methods are (1) cannot locate the first critical mistake (2) incompletion of solution sets. However, in the results part the authors only show that AGPV has good accuracy and good correct coverage, without mentioning that whether AGPV is helpful in terms of first mistake localization and identifying incompletion of solution sets.

  2. Lack of ablation study for each component. AGPV shows multiple components in the entire pipeline. However, the authors do not show the ablation study, so the contribution of each components is not clear. The author should compare AGPV with several counterparts without key components, e.g., the pipeline without math tools, directly verify the claims without listing the key claims first, directly fix the solution without summarization etc.

  3. The comparison is not at fixed budget. AGPV clearly has more LLM calls compared with baselines. The authors should report whether with fixed token budget, AGPV can still outperform baselines, instead of fixing the iteration time as 3. It is possible that with fixed token budget, the baselines can outperform AGPV by doing more iteration

  4. Only two baselines are compared for correction coverage. In Table 2, the authors show that AGPV has better correction coverage. In other words, the error that can be fixed by baselines are largely can be fixed by AGPV. However, in the baselines they only include ProgCo and ProgCo-Py. Actually the coverage comparison should include more baselines, since other methods (such as Self-Refine and Self-Reflection) show comparable performance as ProgCo and ProgCo-Py.

If these four major issues can be addressed, the quality of this paper will be improved significantly.

Minor Issues

  1. Table 2 is a little confusing. First, it mixes the results of all the models together. Second, it only selects the instances that at least one method correct that. I recommend to use 2*2 contingency tables for each model and each comparison. For example, for a specific model and for AGPV-ProgCo comparison, the four cells of the table should show the number of generated solutions that can be corrected by both AGPV and ProgCo, only corrected by AGPV, only corrected by ProgCo, and corrected by neither. Similar for ProgCo-Py. Also, as I mentioned in Comment 4, I also suggest comparing AGPV with Self-Refine and Self-Reflection.

  2. A reference has mistake (the author order of R1-Code-Interpreter). The authors should check whether all the references are correct.

Questions for authors

I already showed my comments in the weaknesses and strengths.

6 comments
AuthorsOpenReview ↗

Response to Reviewer DRQH (1/2)

Thank you for your suggestions. We have added verification examples and correction coverage statistics to show how AGPV locates errors and which incorrect initial answers it can correct. We address each question below.

Q1. How does AGPV locate reasoning errors and detect missing solutions?

1. Locating a reasoning error

AGPV checks the current solution step by step against its verification plan. At the first invalid inference, it identifies the faulty step and gives the corrector the evidence for that judgment. In the following example, AGPV used a specific input to find an error in the derivation of an upper bound.

Example. An error estimation problem from AMO Bench (DeepSeek Flash)

Problem: Given 20252025 real numbers x1,…,x2025x_1,\ldots,x_{2025} in [0,1][0,1], estimate (∑kxk)2(\sum_k x_k)^2 by ∑kf(xk)\sum_k f(x_k). What is the smallest possible worst-case absolute error over all choices of ff?

Original answerAGPV's final answerCorrect answer
1,025,156512,578512,578

Verification: After the first correction, the solution still claimed that its function had an error of at most 1,025,156. On rechecking, AGPV found that setting every xkx_k to 1 gives an error of 2,049,300, exceeding the claimed bound. The corrector redid the derivation and obtained the correct answer, 512,578, which passed the final check.

2. Detecting and recovering missing solutions

We examined the answers from four models on 3 AMO problems that ask for a set of values. Seven initial answers omitted valid solutions. In the first verification round, AGPV identified specific missing values in 5 of those answers. Ultimately, 4 of the 7 included all the correct solutions.

For example, one problem asks for all possible residues modulo 32. The 6 residues in DeepSeek Pro's original answer were all correct, but the list omitted values including 8 and 10. AGPV found the 10 missing residues and returned all 16 even numbers from 0 to 30.

Q2. What do the mathematical tools, verification plan, and feedback summary contribute?

By the time we ran the ablations, DeepSeek had updated its API and the Flash version used in the submission could no longer be selected. The table reports results on the updated DeepSeek Flash with the mathematical tools disabled, the verification plan omitted, or the feedback summary omitted.

MethodAccuracyModel calls per problemTotal tokens (k/problem)
Full AGPV77.1%19.44154.37
D. Disable mathematical tools67.1%6.7154.57
E. Verify directly without first listing key steps71.8%18.33244.66
F. Omit the feedback summary75.9%16.28193.23

(1) Mathematical tools. Full AGPV can use Python to check intermediate results and find and correct calculation errors in the original solution. Variant D, with the mathematical tools disabled, had the lowest accuracy of the three variants.

(2) Verification plan. Full AGPV first lists the key steps and then checks each one. On some problems, variant E skipped the plan and repeatedly read long derivations during correction, increasing mean token usage per problem. Listing the steps in advance may help reduce this repeated analysis.

(3) Feedback summary. Full AGPV summarizes the error location and verification results for the corrector. Variant F reads the original records directly and can still use them to revise the solution. In the table, F achieved accuracy close to that of full AGPV but used more tokens.

Q3. Accuracy under a common token budget

We compared three methods on five datasets under the same token budgets. All methods started from the same initial answers. The budget counts only the input and output tokens used after the initial answer. For each problem, we took the last complete answer within the budget from the recorded run and scored it using the same procedure across methods.

Additional token limit per problemAGPVSelf-RefineProgCo-Py
40,00054.7%59.4%52.9%
90,00065.9%61.8%55.9%
700,00074.1%62.4%62.9%

At 40,000 tokens per problem, Self-Refine had the highest accuracy. AGPV overtook Self-Refine at about 61,000 tokens. As the budget rose from 90,000 to 700,000 tokens, AGPV's accuracy increased from 65.9% to 74.1%, the largest gain over this interval. Figure 7‾\color{#174EA6}{\underline{\textsf{Figure 7}}} in Appendix C.6 of the revised manuscript shows the full budget curves for all three methods.

AuthorsOpenReview ↗

Response to Reviewer DRQH (2/2)

Q4. Correction coverage relative to Self-Refine and Self-Reflection

We extended the comparison to Self-Refine and Self-Reflection, pooling all problems across all evaluated models. For each model, all methods started from the same initial answers. The correction rate is the fraction of initially incorrect answers that became correct.

BaselineAGPV correction rateBaseline correction rateShare of baseline corrections also made by AGPV
Self-Refine38.7%17.3%69.1%
Self-Reflection38.7%10.2%62.5%

AGPV corrected a larger share of initial errors than Self-Refine or Self-Reflection. For each baseline, AGPV also corrected most of the initial answers that the baseline corrected.

Q5. Correction coverage by model

The table shows each method's correction rate on four models: the fraction of initially incorrect answers that were correct at the end. Each row pools one model's results across five datasets. All methods started from the same initial answers.

ModelAGPVProgCoProgCo-PySelf-RefineSelf-Reflection
DeepSeek Flash42.5%11.3%20.0%26.3%11.3%
DeepSeek Pro48.1%6.7%17.3%26.0%5.8%
Qwen 35B22.8%7.6%11.4%8.9%10.1%
GPT-5.6 Luna38.5%3.8%6.9%10.0%13.1%

AGPV had a higher correction rate than all four baselines on each model. For example, among the 80 problems that DeepSeek Flash initially answered incorrectly, AGPV alone corrected 22, ProgCo-Py alone corrected 4, both corrected 12, and neither corrected 42. Table 13‾\color{#174EA6}{\underline{\textsf{Table 13}}} in Appendix C.4 gives the proportions for these four outcomes in the other 15 comparisons.

Q6. Reference check

Thank you for checking our references carefully. We checked the references individually before submitting the manuscript, but we still reversed the first two authors of R1-Code-Interpreter. We apologize for this oversight. In response to your comment, we checked all 37 references again and confirmed that each has a verifiable source.

We have corrected the citation to match the official ICLR 2026 proceedings.

Reviewer_DRQHOpenReview ↗

Clarification of Q1

Thank you for the response. Most of my questions have been addressed except for Q1. For Q1, I think it is better to show statistics instead of real examples. For example, the accuracy of first-error localization among all methods, and also show the fraction of incomplete solution for all methods. I think probably this can be done by analyzing existing responses with a strong LM judge.

Reviewer_SzfsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Agent-Guided Process Verification (AGPV), a multi-agent framework for mathematical self-correction. Given a candidate solution and the problem objective, AGPV constructs an ordered sequence of verifiable key claims and checks them sequentially. When a claim fails, the system uses the verification evidence to guide localized correction of the subsequent reasoning. It then updates or regenerates the key-claim sequence and verification plan based on a comparison of the extracted answers. AGPV also incorporates solution-completeness requirements into process verification to identify omitted solutions or cases.

The paper provides a conditional argument for correctness and completeness, assuming sufficient claim coverage, reliable local verification, and successful verification of all claims. Experiments on five mathematical reasoning datasets comprising 170 distinct problems, using four model configurations, report a mean final-answer accuracy of 63.8% for AGPV, compared with 50.3% for the strongest evaluated baseline, Self-Refine.

Critical strengths

S1. Coherent process verification and localized correction.

AGPV integrates key-claim construction, first-failed-claim localization, tool-supported evidence, targeted correction, and verification-plan adaptation into a coherent workflow (Section 3; Figure 2). Compared with unconstrained natural-language reflection, its feedback is more explicitly grounded in failed claims. Incorporating completeness requirements into verification is also valuable for identifying missing cases that may be overlooked when checking only existing candidate answers.

S2. Substantial empirical improvements across multiple models.

AGPV achieves the highest overall accuracy among the evaluated methods across all four model configurations (Table 1), with a mean improvement of 13.5 percentage points over Self-Refine. Appendix Table 8 reports a paired-bootstrap 95% confidence interval of [9.9, 17.4] percentage points for this difference, conditional on the retained runs. The correction analysis further reports 152 initially incorrect answers successfully corrected, compared with only five initially correct answers degraded (Section 4.3). These findings provide meaningful evidence for the effectiveness of the complete system.

Critical weaknesses

W1. Insufficient evidence of verification reliability for complex reasoning structures.

Although AGPV supports flexible mappings between reasoning steps and key claims, verification still proceeds through an ordered claim sequence (Sections 3.1–3.2). The paper does not adequately demonstrate whether claim generation and subsequent plan updates preserve all necessary verification obligations in solutions involving branching reasoning, shared premises, or interdependent conclusions. This limits confidence in the method's general applicability and practical reliability.

W2. Insufficient isolation of the contributions of core mechanisms.

Despite the substantial overall improvements, the paper lacks systematic ablations of first-failed-claim stopping, completeness checking, and dynamic verification-plan updating. Moreover, using the same maximum number of correction turns does not ensure comparable model calls or inference budgets across methods (Section 4.1; Appendix C.1). Consequently, the individual contributions of these design choices remain unclear.

Questions for authors

Q1. How does AGPV handle nonlinear reasoning dependencies?

Section 3.1 notes that a single reasoning step may yield multiple key claims, while several steps may support a single claim. For solutions involving branching cases or shared intermediate conclusions, how does AGPV ensure that all relevant dependencies are represented? In Section 3.3, when the extracted answer strings match, the system updates rather than regenerates the verification plan. How does this update preserve checks for other branches potentially affected by a local correction? A concrete execution trace or failure-mode analysis would be helpful.

Q2. How reliable are completeness verification and the conditional guarantee in practice?

The argument in Appendix B.1 assumes sufficient coverage of completeness requirements and reliable local verification, yet generated key claims may omit necessary cases or obligations. Can the authors separately evaluate the detection of missing solutions or branches? Additionally, Algorithm 1 may return the latest corrected candidate without further verification when the iteration limit is reached. What fraction of outputs fall into this category, and how accurate are they?

Q3. Which core design choices are necessary for the observed improvements?

Sections 4.1 and C.1 indicate that identical maximum correction-turn counts do not equalize actual computational budgets, while Appendix C.5 provides only partial cost accounting. Can the authors provide ablations of first-failed-claim stopping, completeness checking, and verification-plan updating, along with comparable statistics on model calls, input/output tokens, and tool usage? This would help clarify the extent to which the observed improvements are attributable to the proposed mechanisms.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Szfs (1/3)

Thank you for your suggestions. We have added verification examples, ablation results, and computational costs. We address each question below.

Q1. Verifying branches and updating the verification plan

AGPV generates a verification plan from the problem conditions and the current solution. When several branches share a premise, AGPV can reuse the verification result for that premise and then check each branch's conclusion separately. If the problem asks for all solutions, AGPV also needs to check whether the branches together cover the full solution set.

1. Checking whether all solutions have been found

When solving an equation, AGPV can give the original equation and its domain to a mathematical tool and compare the complete solution set returned by the tool with the solutions listed in the candidate answer.

Example 1. Solving an equation with SymPy

Problem: Find all real solutions to x2=4x^2=4.

  • Original answer: {2}\{2\}

  • SymPy result: {−2,2}\{-2,2\}

  • Correct answer: {−2,2}\{-2,2\}

Verification: Pass the original equation x2=4x^2=4 to SymPy and specify the real domain.

from sympy import Eq, S, solveset, symbols
x = symbols("x")
solveset(Eq(x**2, 4), x, domain=S.Reals)  # Output: {-2, 2}

SymPy obtains both real roots directly from the original equation. Comparing them with the original answer lets AGPV identify the missing root, −2-2.

2. Rechecking affected branches when the final answer stays the same

If verification fails but the final answer stays the same after correction, the original plan may have missed the faulty step. AGPV updates the plan using the verification feedback and the revised solution. If another round remains, verification starts again at the first item in the updated plan. The following example shows a case where the final answer stays the same.

Example 2. An AIME 2024 run

Problem: Once the prime has been identified as 17, find the smallest positive integer mm for which 172=28917^2=289 divides m4+1m^4+1.

  • Original answer: 110. The four branches gave 110,138,151,179110,138,151,179.

  • AGPV's final answer: 110. The corrected branches gave 110,134,155,179110,134,155,179.

  • Correct answer: 110.

Verification: The original solution considered the four possible residues of mm modulo 17 separately. AGPV found that 138 and 151 failed the divisibility condition and replaced them with 155 and 134, respectively. The minimum remained 110. In the next round, the verifier rechecked each of the four branches under the updated plan. It enumerated one full period modulo 289 in Python and confirmed all four values. The original final answer was correct, but two intermediate calculations needed correction.

3. When verification can still miss solutions

We found an AMO Bench problem that asks for all possible residues. AGPV recovered some missing residues by enumeration, but both verification rounds used the same range, so some residues remained undiscovered.

Example 3. An AMO Bench run

Problem: As nn ranges over the positive integers, what residues can ⌊n!(n+1)(n+2)⌋\left\lfloor \frac{n!}{(n+1)(n+2)}\right\rfloor take modulo 32?

  • Original answer, 7 residues: {0,2,4,6,12,16,26}\{0,2,4,6,12,16,26\}

  • AGPV's final answer, 11 residues: {0,2,4,6,12,14,16,20,24,26,30}\{0,2,4,6,12,14,16,20,24,26,30\}

  • Correct answer, 16 residues: {0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30}\{0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30\}

Verification: In the first round, AGPV enumerated n=1n=1 to 5050 in Python and added four residues missing from the original answer: 14, 20, 24, and 30. In the second round, the verifier enumerated the same range. The 11 residues it found matched the revised answer, which passed verification.

Why verification missed the remaining residues: Both rounds stopped at n=50n=50. Evaluating the original expression for just one more term, n=51n=51, gave 28, which was missing from the answer. Verification correctly computed the residues within the chosen range but did not establish that the range covered every possible residue. Compared with the reference answer, AGPV still missed 8, 10, 18, 22, and 28.

AuthorsOpenReview ↗

Response to Reviewer Szfs (2/3)

Q2. Detecting and recovering missing solutions, and accuracy at the round limit

1. Can AGPV detect and recover missing solutions?

We examined 12 initial answers from 4 models on 3 AMO Bench problems. Seven listed specific integers but omitted some valid values. Of these 7 initial answers, AGPV identified specific missing values in 5 during the first round and ultimately produced complete, correct answers in 4.

The residue problem in Q1 has 16 valid residues. We compare DeepSeek Pro and Qwen 35B on that same problem below.

  • Example 1. DeepSeek Pro recovered all the residues. Its original answer listed 6 residues. AGPV found the 10 missing ones, and its final list of 16 matched the reference answer.

  • Example 2. Qwen 35B recovered some of the residues. This is the run shown in Q1. Its original answer listed 7 residues. AGPV added 4 and returned 11, still missing 5. Both verification rounds stopped at n=50n=50, although n=51n=51 already gives the missing residue 28.

2. How often are answers returned at the round limit, and how accurate are they?

The table groups the 680 outputs from the four models by how verification ended. The second column gives each group's share of all outputs, and the third gives accuracy within that group.

Return conditionShare of outputsAccuracy within group
First check passed; original solution returned52.1%75.4%
First check failed; later check passed35.0%62.6%
Round limit reached; answer returned after the last check failed12.9%20.5%

Accuracy was highest for solutions that passed the first check, followed by those that passed after correction. Solutions that still fail verification at the round limit are also returned, and this group had the lowest accuracy. Incorrect final answers in this group accounted for 10.3% of all outputs. Thank you for pointing this out. We will investigate how to improve the accuracy of answers returned at the round limit.

AuthorsOpenReview ↗

Response to Reviewer Szfs (3/3)

Q3. Effects of three design choices and their computational costs

1. Ablations of the three design choices

The table reports three ablations on DeepSeek Flash$^{*}$.

MethodAccuracyModel calls per problemTotal tokens (k/problem)
Full AGPV77.1%19.44154.37
A. Continue past the first error74.1%17.96226.95
B. Remove the explicit prompt to check for omissions72.4%16.34198.14
C. Keep the first-round plan fixed76.5%16.03128.38

∗^{*} DeepSeek Flash denotes the version before the update. DeepSeek Flash$^{*}$ denotes the updated version.

(1) Moving directly to correction at the first error can save input tokens. AGPV ends verification for that round at the first error and need not read the remaining derivation. Without early stopping, it used an additional 72.58k tokens per problem, including 69.24k additional input tokens.

(2) Explicitly checking for omissions helps verify the solution set. On a problem asking for all integer solutions, full AGPV returned the correct set, {1, 2, 3}. Without the prompt to check for omissions, the answer also included 4 and 5.

(3) Updating the verification plan had little effect on overall accuracy on these problems. AGPV updates the plan so that the next round checks the revised reasoning. Accuracy was 76.5% when the first-round plan was held fixed.

2. Computational costs by method

AGPV and the iterative correction baselines start from the same initial answers. The table reports accuracy and mean cost per problem on five datasets with DeepSeek Flash.

MethodAccuracyModel callsInput tokens (k)Output tokens (k)Python runsTime (min)
Shared initial answer52.9%1.862.973.5400.57
Reflex60.0%3.0010.339.0400.61
Self-Refine61.8%6.0023.7712.1800.92
Self-Reflection55.3%3.0011.5510.6600.77
CheckList55.3%4.0011.228.7400.65
CoT-Tool54.1%3.000.5810.2900.72
ProgCo58.2%7.8245.3111.9301.07
ProgCo-Py62.4%48.5516,945.8858.6139.7931.76
AGPV72.4%14.7180.338.904.062.07

(1) Compared with standard feedback-based correction methods, AGPV used more input tokens and achieved higher accuracy. For example, AGPV outperformed Self-Refine, the most accurate of these baselines, by 10.6 percentage points. The additional input was mainly used for stepwise verification: the model reads the reasoning steps and tool outputs before revising the solution.

(2) Compared with ProgCo-Py, which also uses Python, AGPV achieved 10.0 percentage points higher accuracy with nearly 90% fewer Python executions per problem. AGPV moves to correction after the first failed step. When the same tool request recurs within a problem, AGPV reuses its previous successful result, avoiding 17 duplicate Python executions. At the soft limit on consecutive tool calls, the system prompts the model to answer using the results already available. At the hard limit, it ends the current tool call.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Manuscript revisions

For ease of reading, we highlight the key points in bold in our responses. In the revised manuscript, the main additions to the text and appendix are marked in blue.

The supplementary material includes the data for the additional experiments reported in our responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.