acceptodds
Under review as a conference paper at ICLR 2027

Revisiting Mathematical Reasoning Verification with Deep-Research-Grounded Trajectory Distillation

Abstract

Large language models (LLMs) have shown strong mathematical reasoning capabilities, yet reliably verifying existing reasoning traces remains challenging. A reliable verifier should detect genuine errors, localize the authoritative first error step, and avoid unsupported judgments on valid reasoning. We propose Deep-Research-Grounded Trajectory Distillation (DRGTD), a framework that combines structured verification supervision with conditional localization optimization. We first construct a multi-agent teacher–reviewer trajectory-construction pipeline centered on a Deep-Research Teacher. The Teacher performs source-grounded semantic analysis and selective external mathematical checks to produce structured verification trajectories. Fail-closed trajectory validation then retains supported information and compiles it into Decision, First-Error, Process, and Unified-Deployment supervision. We next apply two-stage trajectory distillation. Phase 1 performs multi-view capability learning, while Phase 2 aligns the learned behavior with the natural deployment interface. Finally, we freeze the SFT reasoning-trace-level CLEAN/ERROR decision gate and apply frozen-gate RL only to conditional first-error localization using pairwise hard negatives and frozen-reference KL regularization. On the primary Qwen2.5-Math-7B backbone, Strict Verification Accuracy improves from 29.33% for Base to 35.33% after SFT and 37.00% after RL. BaseSFT also improves Strict Verification Accuracy and Exact FE across 10 open-weight backbones. On three representative backbones, Pairwise PG + KL achieves higher Exact FE, FE|Detected, and Strict Verification Accuracy than PPO and Bandit-OPD. Teacher-side confirmatory evaluation shows a directional but not statistically significant improvement over native verification. Overall, our results support a controlled path from Deep-Research teacher verification and validated trajectory distillation to frozen-gate conditional localization optimization for compact mathematical reasoning verifiers.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Language Model, Agent, Deep Research, Trajectory Distillation, Reinforcement Learning, Mathematical Reasoning.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JPfVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper works on the mathematical reasoning verification. It designed a Deep-Research-Grounded Trajectory Distillation workflow followed by RL to increase the verification ability of the base model. In its main experiment, they used trajectories generated by DeepSeek-V4-Pro to distill Qwen2.5-Math-Instruct and reported certain performance gains.

Critical strengths

None

Critical weaknesses
  • Poor presentation makes it hard to understand the method and key insights in this work.
  • An important baseline is omitted in this work. Since this paper is using a distillation method with a specially designed workflow to increase the verification ability of smaller models, it is not surprising that in Table 3 all training backbones received a performance increment. But at least we should include a comparison with direct SFT distillation on the direct generation trajectory of the teacher (DeepSeek-V4-Pro) without any agentic workflow, or we will not be able to determine how much of the performance gain can be attributed to the framework proposed in this work. This work did not compare their method with others as well.
  • The amount of experiments presented in this paper is far too limited to support the main conclusions.
  • This work provides little insights or valuable discoveries for future works.
Questions for authors
  • In Table 3, all models evaluated obtained exactly the same score on CLEAN and Detection under RL and SFT + RL setting. Why would this happen? I believe this is a severe data error that will reduce the overall soundness of the paper.
Reviewer_YHhUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies whether a model can judge a mathematical solution’s correctness and locate its first erroneous step. It proposes distilling a tool-assisted teacher’s verification outputs into smaller models through filtered, structured supervision and supervised fine-tuning, followed by reinforcement learning for error localization while keeping correctness decisions fixed. Evaluation on PRM300 shows that SFT improves verification accuracy across ten student backbones. For the main Qwen2.5-Math-7B-Instruct model, strict verification accuracy increases from 29.33% to 35.33% after SFT and 37.00% after the final stage. Ablations examine supervision formats, deployment alignment, and localization optimization; teacher improvements are not statistically significant, and RL gains vary across backbones.

Critical strengths

Supervised distillation improves both strict verification accuracy and exact first-error localization across all ten evaluated student backbones on PRM300. This consistency provides evidence that the training pipeline’s benefits are not limited to a single model.

Critical weaknesses

The paper does not clearly establish the benefit of its proposed pipeline over ordinary teacher-to-student distillation. While the SFT results show that students improve after training, they do not isolate the added value of the tool-assisted teacher and trajectory validation. Under matched training data and budgets, how much does the proposed approach improve over distillation from the Native Teacher? This comparison is important for determining whether the additional complexity is justified.

The benefit of the Deep-Research Teacher is insufficiently established. Its improvement over the Native Teacher is modest and not statistically significant. More importantly, the paper does not directly compare students distilled from the two teachers under matched training conditions. Such a comparison would clarify whether the additional teacher-side complexity produces better students. The reported supervision-view ablations do not resolve this question.

Questions for authors

Have you compared students distilled from the Native Teacher and the Deep-Research Teacher using the same training examples, supervision formats, and training budget? If so, what are the results? This comparison would materially affect my assessment of whether the proposed teacher-side complexity is justified.

What intended use motivates requiring exact first-error localization? Is there evidence that improvements in this metric lead to more effective solution correction, answer selection, or reasoning-model training? Please clarify whether such benefits are demonstrated in this work or remain potential future applications.

Reviewer_zPAqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DRGTD for mathematical reasoning verification. It uses a tool-augmented deep research teacher to generate verification trajectories, filters them, and distills them into smaller verifiers through SFT. It then freezes the error detection gate and optimizes first-error localization.

Critical strengths
  1. The paper addresses an important problems, detecting reasoning errors and locating the first incorrect step, in math questions.

  2. SFT consistently improves verification performance across ten backbones.

  3. The frozen-gate setting helps separate improvements in localization from changes in error detection.

Critical weaknesses
  1. The novelty over previous verifier distillation methods is unclear. The main difference seems to be the Deep-Research teacher and how its trajectories are processed.

  2. There is no controlled comparison between students trained on native-teacher and Deep-Research trajectories. It is therefore unclear whether Deep Research is necessary for the gains.

  3. Comparisons with existing trained verifiers, like ThinkPRM, GenPRM and Math-Minos are limited. Most experiments compare against the base model or variants of the proposed method.

  4. Although the paper includes some ablations, it does not isolate the contribution of each key component. For example, there is no comparison with and without evidence binding, D and F supervision are added together, and the frozen gate is not compared with joint optimization of detection and localization.

  5. The additional RL gains are small and inconsistent across backbones. Larger evaluations and multiple seeds would help establish their reliability.

Questions for authors
  1. What is the main difference from previous verifier distillation methods beyond using a Deep-Research teacher?

  2. Could you compare students trained on native-teacher and Deep-Research trajectories under the same training budget to isolate the benefit of Deep Research?

  3. How does DRGTD perform compared with existing verifiers such as ThinkPRM[1], GenPRM[2], and Math-Minos[3] under the same evaluation setting?

  4. Could you provide ablations for evidence binding, D and F supervision separately, and frozen-gate versus joint optimization to clarify each component’s contribution?

  5. Are the RL gains consistent across multiple seeds and larger evaluation sets, given the small improvements and mixed results across backbones?

  6. The Primary Area choses as applications to computer vision, audio, language, and other modalities. That looks very off to me. What's the reason of this?

[1] Muhammad Khalifa et al. Process Reward Models That Think. arXiv:2504.16828, 2025. https://arxiv.org/abs/2504.16828

[2] Jian Zhao et al. GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning. arXiv:2504.00891, 2025. https://arxiv.org/abs/2504.00891

[3] Bofei Gao et al. LLM Critics Help Catch Bugs in Mathematics: Towards a Better Mathematical Verifier with Natural Language Feedback. Findings of ACL, 2025. https://aclanthology.org/2025.findings-acl.753/

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.