- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Agent-Guided Process Verification (AGPV) for mathematical self-correction. With multiple agents, AGPV detects and correct the first error in the key claims. The experiments prove its effectiveness on multiple math benchmarks.
Critical strengths
The proposed method is explained clearly.
The experiment is comprehensive. By comparing with multiple baseline methods on multiple benchmarks, the effectiveness of the proposed method is proven.
Critical weaknesses
-
This framework uses multiple agents, which is likely to cost a lot more tokens. It is better to comprehensively compare the cost across all the methods. In Line472, the author claims that “Full per-call usage is unavailable across AGPV, ProgCo-Py, and Self-Refine”. Why not available?
-
The prompt for each agent is missing in the paper. It would help understand the details if the prompt is written in the paper.
Questions for authors
In Table 1, Qwen3.6 35B-A3B-FP8 performs better than other models using Initial method, which is surprising and counterintuitive as DeepSeek V4.1-Flash should be much better than Qwen3.6 35B-A3B-FP8. How to explain this?
Response to Reviewer maWx
Thank you for your suggestions. We have added a comparison of computational costs across methods and initial-answer results for the two models. We address each question below.
Q1. Computational costs by method
We measured the mean cost per problem for each method using DeepSeek Flash on five datasets.
(1) Compared with standard feedback-based correction methods, AGPV used more tokens and achieved higher accuracy. For example, AGPV outperformed Self-Refine by 10.6 percentage points. The additional input was mainly used for stepwise verification: the model reads the reasoning steps and tool outputs before revising the solution.
(2) Compared with ProgCo-Py, which also uses Python, AGPV achieved 10.0 percentage points higher accuracy with nearly 90% fewer Python executions per problem. AGPV moves to correction after the first failed step and skips the remaining checks. When the same tool request recurs within a problem, AGPV reuses its previous successful result, avoiding 17 duplicate Python executions in this experiment. At the soft limit on consecutive tool calls, the system prompts the model to answer using the results already available. At the hard limit, it ends the current tool-call loop.
Q2. Prompts for the agents in Figure 2
The supplementary code submitted with the paper includes the prompts for all six agents in Figure 2. The text immediately before Table 3 in Appendix A of the revised manuscript gives the location of the full prompt file. The table below lists the core instructions for each agent.
Q3. Initial-answer accuracy of Qwen and DeepSeek Flash
We compared the models' initial answers using the same test problems and scoring procedure. Overall accuracy is computed across all test problems, with equal weight per problem.
Qwen and DeepSeek Flash each led on two datasets and tied on the fifth. Qwen's initial-answer accuracy on AMO Bench was 16.0 percentage points higher. In problem counts, Qwen's extra correct answers on AMO Bench exactly offset DeepSeek Flash's combined lead on its two stronger datasets. The overall gap comes from HMMT 2025, leaving Qwen ahead by 0.6 percentage points.