acceptodds
Withdrawn from ICLR 2027

Judging the Path, Not Just the Answer: Turn-Level Verifiers for Deep Research Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

19 views

Abstract

Autonomous deep research agents tackle complex, open-ended questions by sequentially reasoning and interacting with external environments across extended horizons. The prevailing reinforcement learning paradigm relies on terminal outcome rewards and encounters a fundamental turn-level credit assignment bottleneck. For failed trajectories, terminal penalties uniformly suppress all intermediate actions without distinguishing productive exploratory steps from genuine errors. On challenging queries where all rollouts fail, the learning signal vanishes. While process-level reward paradigms offer finer-grained feedback, they typically evaluate actions against ground-truth answers, limiting their ability to credit intermediate progress along unsuccessful paths and precluding their deployment during test-time search. To this end, we propose TPV (Turn-level Process Verifier), an unprivileged process verifier that evaluates each intermediate action purely from the interaction context. TPV is trained to predict turn-level grades assigned by an offline, privileged teacher judge that evaluates each action independently of final trajectory success. Empirical evaluations show that TPV reliably discriminates informative from redundant turns even inside failed rollouts, outperforming outcome-based baselines and transferring across diverse model scales. Consequently, our unified verifier serves a dual purpose: it delivers dense, fine-grained credit assignment during reinforcement learning, bypassing the zero-gradient dilemma on all-fail queries, and acts as an effective trajectory reranker at inference time. Extensive experiments on five challenging deep research benchmarks demonstrate that integrating TPV into RL training consistently boosts task success rates over competitive baselines, while deploying it as a test-time reranker yields substantial additional accuracy gains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
process reward model, deep research agents, credit assignment, test-time scaling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eavwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes TPV, a turn-level process verifier for deep research agents. An offline privileged teacher with access to ground truth assigns A/B/C/D grades to search and visit turns. An unprivileged student verifier is then trained to predict these grades from interaction context alone. The frozen verifier serves two purposes: providing dense turn-level rewards during GRPO training and reranking sampled trajectories at test time. Experiments cover five deep research benchmarks and compare TPV against ORM, IGPO, and a uniform-reward ablation, with additional analyses of verifier scaling, cross-architecture transfer, and reranking aggregation.

Critical strengths
  1. The paper addresses a relevant and practically motivated problem in turn-level credit assignment. It identifies a real limitation in multi-turn deep research agents: terminal outcome rewards cannot easily distinguish productive exploration from unproductive or harmful intermediate actions, and all-fail rollout groups may provide limited learning signal. TPV’s turn-level process supervision offers a plausible direction for mitigating this issue.
  2. The proposed framework is reasonably clear, and the empirical results are somewhat encouraging. Separating privileged label construction from unprivileged verifier deployment is a sensible design choice, and reusing the same frozen verifier for both RL training and test-time reranking is a potentially useful contribution. The evaluation across five benchmarks, together with verifier scaling and reranking analyses, provides some initial evidence of practical potential.
Critical weaknesses
  1. The cross-architecture generalization claim is not supported at the policy level.

RQ4 asks whether the framework generalizes across policy architectures and verifier scales. However, Section 3.5 only replaces the student verifier's backbone from Qwen3-8B to Llama-3.1-8B and compares verifier macro-F1 (0.643 vs. 0.648). The policy remains Qwen3-4B-Thinking throughout the experiments.

Therefore, the current experiment demonstrates that turn-level discrimination can transfer across verifier model families, but it does not establish that TPV as an RL training method generalizes across policy architectures. The conclusion that performance stems from process supervision rather than biases tied to a particular backbone is consequently broader than the evidence supports. Different policies may respond differently to process rewards, and TPV's improvement over ORM could vary substantially across architectures.

This gap is particularly important because cross-policy generalization is explicitly included in the research questions, and Section 5 concludes that TPV "scales effectively across policy architectures." The claim would be much better supported by evaluating at least one additional policy architecture under comparable SFT, ORM, and TPV settings.

  1. An internal inconsistency in the reported GAIA results makes the magnitude of the main performance gain unclear.

In Table 1, the TPV row reports GAIA accuracies of 76.19%, 57.58%, and 31.58% for L1, L2, and L3, respectively. According to Table 4, these levels contain 42, 66, and 19 questions, for a total of 127 questions. The reported level-wise accuracies therefore imply 32, 38, and 6 correct answers, respectively, yielding an overall accuracy of 76/127 = 59.84%. However, Table 1 reports 53.54%, corresponding to 68/127 correct answers. Section 3.2 repeats the reported overall figure of 53.54%.

The discrepancy should be checked against the original evaluation results and the inconsistent entries corrected. While this does not by itself establish that the other benchmark results are incorrect, it leaves the reported GAIA performance and the magnitude of TPV's improvement over ORM uncertain.

  1. The necessity and practical benefits of teacher-to-student distillation are not sufficiently established.

Appendix D.2 reports a turn-level AUC of 0.820 for the teacher without ground-truth access, compared with 0.777 for the student verifier. This raises an important question about the motivation for distillation: what does the student provide beyond what could be obtained by deploying the teacher in its ground-truth-free configuration?

The comparison requires clarification. In particular, the paper should specify how the AUC scores are computed and whether the teacher and student use comparable grade-to-score mappings. A difference in offline AUC alone does not establish which model is more effective for the intended applications. Comparing the two models as downstream RL reward providers and test-time rerankers would be more informative. If deployment efficiency motivates distillation, the student's advantages in inference cost, memory, or throughput should also be quantified.

Questions for authors
  1. GAIA result consistency: Table 1 appears to be internally inconsistent. The TPV L1/L2/L3 accuracies (76.19% / 57.58% / 31.58% over 42 / 66 / 19 questions, respectively) imply 76/127 = 59.84%, whereas the reported overall accuracy is 53.54% (= 68/127). Could you check the original evaluation results and correct the inconsistent entries? Section 3.2 repeats the same overall figure.
  2. Teacher-to-student distillation: Appendix D.2 reports a higher turn-level AUC for the teacher without ground-truth access than for the student verifier (0.820 vs. 0.777). Could you clarify how these AUC scores are computed and whether both models use comparable grade-to-score mappings? More importantly, what motivates distillation in this setting? Could you compare the two models' downstream performance as RL reward providers and test-time rerankers, and quantify the student's efficiency advantage?
  3. Cross-policy generalization: Could you evaluate TPV with at least one additional policy architecture, reporting results for SFT, ORM, and TPV? Comparing TPV's improvement over ORM across policy architectures would directly test the cross-policy generalization claim in RQ4. Alternatively, the authors should narrow the corresponding claims to verifier-backbone transfer.
Reviewer_xmBLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TPV for turn-level credit assignment in deep research agents. A teacher with access to ground-truth answers, but blinded to final trajectory outcomes, labels search and visit turns. Then a context-only student learns to predict these labels. The frozen verifier provides process rewards alongside outcome rewards during GRPO training and supports test-time Best-of-N trajectory reranking. Reported results show improved policy performance across five deep research benchmarks and further gains through test-time trajectory reranking.

Critical strengths
  1. This paper addresses a practical credit assignment problem without requiring ground-truth answers or branch rollouts at deployment. And I think using the same verifier for training and reranking is a coherent design.
  2. Uni-TPV provides a useful control for credit placement. Grade distributions within successful and failed trajectories, together with alignment against human-annotated evidence, support the local utility of process labels.
Critical weaknesses
  1. TPV's GAIA breakdown implies 59.84% overall accuracy, not 53.54% (Table 1; Appendix A.1). Appendix B.2 reports 867 trajectories with final-answer turns but only 843 excluded turns, including final answers. These discrepancies may require clarification to establish my confidence in the reported results.
  2. Uni-TPV tests credit placement, but don't have matched rule-only and learned outcome-verifier baselines. It is difficult to assess the learned verifier's incremental benefit and the computational trade-off introduced by turn-level scoring without these controls or runtime/cost comparisons.
Questions for authors
  1. Please clarify the GAIA results and evaluation-set accounting mentioned above and provide corrected counts. Do the main comparisons still hold after correction?
  2. Were splits separated by question, and was overlap with evaluation benchmarks checked?
  3. How much computational overhead does TPV's turn-level scoring add to RL training, and do its gains over ORM and IGPO persist under a matched training compute budget?
  4. I would be interested in a matched rule-only control, as well as downstream results comparing students trained on privileged and unprivileged teacher labels.
Reviewer_djcUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the reward design of training deep research agents and identifies turn-level credit assignment as an important direction. To relieve the reliance on ground-truth answers in current process-level reward paradigms, the authors design a Turn-level Process Verifier (TPV), where the reward is provided by a trained verifier distilled from a teacher judge with oracles. Experiments on 5 benchmarks (GAIA, BrowseComp-plus, BrowseComp, xBench, DeepSearchQA) and seven baselines (including GRPO, ORM, IGPO) show the gain of the proposed method. Further analysis suggests that (1) the verifier quality scales with the model size; (2) the trained verifier can serve as a useful reranker at test-time.

Critical strengths
  1. This paper is well-written and easy to follow

  2. The experiment design overall makes sense and is clearly divided into multiple research questions (performance, turn quality, judge, and test-time re-ranking, etc.)

Critical weaknesses
  1. This paper has incremental methodological contribution: a teacher with oracles, a trained student verifier, reward combination for RL, and test-time reranking are all standard pipelines for agent training. The paper serves only as an empirical application and verification of the pipeline on deep research agents.

  2. The experiments do not show clear strength of the proposed method over the baselines. Compared with IGPO, the gains are modest on xBench, DeepResearchQA, and BrowseComp. Besides, the re-ranking experiments (Figure 4) are only compared with majority voting, instead of strong baselines such as trajectory-level judges, which can help identify the contribution of the proposed design.

  3. The experiments are not enough. Although Section 3.5 evaluates different verifier sizes and architectures, the policy optimization experiments are limited to Qwen3-4B, which can provide limited insight into its generalizability to other model/data scales. The proposed method is also not well ablated – TPV combines the components of teacher annotation, verifier training, Turn-level reward, etc without testing their individual effectiveness.

Questions for authors
  1. In Table 1, the TPV accuracies for GAIA L1/L2/L3 imply an overall accuracy of approximately 59.84%, using the subset sizes of 42/66/19 reported in Appendix A.1. However, the reported overall accuracy is 53.54%. Could the authors clarify this discrepancy and confirm the correct results?

  2. Could the authors include the performance of a stronger, larger agent model as a reference for the experimental setting in the paper?

1 comment

Thank you for suggesting stronger reranking baselines and for flagging the inconsistency in the GAIA results.

  1. Reranking baselines (Figure 4)

We compared all methods on the same 128 BrowseComp-Plus questions, with eight fixed candidates per question. No reranker had access to the reference answers.

RerankerCorrect / 128Accuracy
Rules-only50/12839.06%
Majority voting52/12840.63%
Qwen3-4B-Thinking-2507, answer-only57/12844.53%
TPV-8B63/12849.22%
DeepSeek V4.1 Flash, answer-only66/12851.56%
Qwen3-4B-Thinking-2507, whole trajectory68/12853.13%
DeepSeek V4.1 Flash, whole trajectory78/12860.94%

We focus on the answer-only comparison because it directly tests whether TPV's gains in Figure 4 can be recovered by strengthening the original majority-vote baseline. The answer-only selectors use the same eight final answers as majority voting and replace vote counts with the model's judgment given the question. TPV improves over majority voting by 8.59 percentage points and over answer-only Qwen by 4.69 points, and comes within 2.34 points of answer-only DeepSeek. The whole-trajectory judges provide a complementary comparison by evaluating each candidate's final answer alongside the evidence from its full rollout.

TPV also outperforms the rules-only control by 10.16 percentage points. This ablation holds the candidate pool, tool turns, and mean pooling fixed and measures what the learned turn scores add beyond the novelty and format rules.

  1. GAIA consistency (Table 1 and Section 3.2)

We ran multiple evaluations. The originally reported overall score of 53.54% (68/127) came from an earlier complete evaluation, and the higher per-level scores, which imply 59.84% overall, came from another evaluation. The inconsistency arose because we combined figures from different runs in the same row.

We will update Table 1 and Section 3.2 using all four figures from the latest complete TPV run: 73/127 (57.48%) overall, with 30/42 (71.43%) on L1, 37/66 (56.06%) on L2, and 6/19 (31.58%) on L3.

Comments on the paper

1 comment
Area_Chair_vjvcOpenReview ↗

Discussion phase ending soon

Dear Reviewers,

As the discussion phase is coming to an end, could you please take a moment to review the authors' responses and finalize your ratings?

If you have any remaining concerns or questions, please feel free to raise them with the authors before the discussion closes.

Thank you very much for your time and support!

Best regards, Xiaojuan

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.