- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the challenge of providing turn-level supervision for reinforcement learning (RL) of deep-research agents on open-ended tasks without canonical answers. The proposed approach uses task-specific requirements as a shared reference for both final-answer evaluation and intermediate process supervision. In particular, it introduces a credit assignment mechanism that measures the incremental support contributed by each research step towards satisfying these requirements. The method is evaluated on four benchmarks, including DeepRubric as the in-domain benchmark and three others for out-of-domain generalisation.
After rebuttal: The authors responses partially address my concerns in review, and I have updated my overall score accordingly.
Critical strengths
S1. Comprehensive baseline comparisons. The evaluation covers a broad range of baselines, including proprietary frontier models, open-source models, conventional RAG approaches, and 8B variants of both open-search and deep-research agents. This provides a useful basis for assessing the proposed method against different approaches to research-agent design and training.
S2. Useful ablation studies. The ablation studies in Tables 2 and 3 help examine the contributions of different components of the proposed approach, providing additional evidence beyond the overall performance comparisons.
Critical weaknesses
W1. Insufficiently differentiated novelty. The paper should more clearly distinguish its contributions from existing work. Rubric-based rewards have already been explored in RaR, DeepRubric, and DR Tulu, including applications to deep-research agents. Similarly, using intermediate signals to guide RL training for research agents is not new, as demonstrated by InfoReasoner. The key question is therefore whether the proposed requirement-grounded incremental credit assignment introduces a fundamentally different supervision signal or mainly combines existing ideas. A more explicit conceptual and empirical comparison with these closely related approaches is needed.
W2. Reliability and computational cost of credit assignment. The proposed method relies on LLM judges to estimate accepted support increments (Equation 4), making the quality of turn-level supervision dependent on judge reliability. The paper provides insufficient analysis of judge consistency, potential evaluation biases, and how erroneous intermediate judgments may affect RL training. In addition, the method introduces substantial computational overhead. Table 7 reports a latency exceeding 23 seconds, compared with approximately 2.4 seconds for GRPO, representing nearly a tenfold increase. This raises concerns about its practical scalability and whether the performance improvements justify the additional training cost.
Questions for authors
Q1. Novelty relative to prior work: What is the key methodological difference between the proposed requirement-grounded credit assignment and existing rubric-based reward approaches such as RaR, DeepRubric, and DR Tulu? How does it differ from intermediate reward mechanisms such as InfoReasoner? In particular, is the novelty primarily in the reward formulation, the use of task requirements, or the estimation of incremental support?
Q2. Judge reliability: How sensitive is the method to inconsistencies or errors in the LLM judge used to estimate accepted support increments? Have the authors measured agreement across different judges, repeated evaluations, or human annotations? Could judge disagreement or uncertainty be incorporated into the credit assignment mechanism to reduce noisy supervision?
Q3. Computational efficiency: Given the nearly tenfold latency increase over GRPO reported in Table 7, how does the method compare under matched computational budgets? Could the authors report performance against total training time, GPU-hours, or judge calls, and investigate whether less frequent or selective intermediate evaluations could achieve similar gains at lower cost?
6 comments
Response to Reviewer edtQ (comparison with prior work)
Q4:
A more explicit conceptual and empirical comparison with these closely related approaches is needed.
A4: The following table summarizes the conceptual differences in supervision source and the quantity being rewarded.
| Method | Open-ended generation | Process rewards | Key distinction |
|---|---|---|---|
| Mathematical PRMs: Let's Verify Step by Step; Math-Shepherd | No: mathematical reasoning | Yes | Train PRMs using human step labels or automatically derived labels based on reference answers, respectively. |
| RaR | Yes: includes open-ended medical responses | No | Uses task-specific rubrics to reward final responses. |
| DeepRubric | Yes | No | Constructs evidence-tree-grounded rubrics for final-report rewards. |
| DR Tulu | Yes | No | Evolves final-report evaluation rubrics using retrieved evidence and on-policy responses. |
| InfoReasoner | No: QA with reference answers | Yes | Scores retrieval through changes in the probability of the correct-answer class (Eq. 17), then aggregates these scores into a trajectory reward. |
| RubricEM | Yes | Yes | Compares trajectories for the same question to generate and evolve stage-specific scoring rubrics for stage-level rewards. |
Here, “open-ended generation” refers to tasks whose output quality cannot be determined solely by a single reference answer; “process rewards” assess the quality or contribution of individual steps or stages. The InfoReasoner entry describes its Eq. (17); its reference-free semantic-entropy formulation is discussed in A2.
Dr.Credit directly reuses final-report rubrics to assess intermediate evidence contributions, with per-rubric support histories for Visit. This provides process supervision for open-ended tasks without a canonical answer or separately generated process rubrics.
Response to Reviewer edtQ (process-judge reliability and training sensitivity)
Q5:
Have the authors measured agreement across different judges, repeated evaluations, or human annotations?
A5: Yes. We conducted same-model repeated evaluation and cross-judge evaluation on 256 training query groups, comparing the resulting credit levels and advantages with those from the original training assessments. Qwen3.6-35B-A3B is the original process judge; its row measures repeatability, while DeepSeek-Flash and GLM-5.3 measure cross-model agreement.
We assess both the judgments and the training signals derived from them:
- Level agreement: the percentage of re-evaluated credit levels matching the original levels.
- Process cosine: cosine similarity between the original process advantages and those recomputed from the re-evaluated credits using the same group normalization.
- Process sign agreement: the percentage of process advantages whose signs agree before and after re-evaluation.
- Fused sign agreement: sign agreement after combining the recomputed process advantages with the unchanged original outcome advantages.
| Re-evaluation judge | Level agreement (%) | Process cosine | Process sign agreement (%) | Fused sign agreement (%) |
|---|---|---|---|---|
| Qwen3.6-35B-A3B | 95.4 | 0.968 | 97.4 | 97.1 |
| DeepSeek-Flash | 93.5 | 0.943 | 93.8 | 93.2 |
| GLM-5.3 | 89.7 | 0.912 | 92.3 | 92.6 |
Same-model re-evaluation yields 95.4% level agreement and 0.968 process-advantage cosine similarity. Across the two alternative judges, level agreement is 89.7–93.5%, process-advantage cosine similarity is 0.912–0.943, and process and fused sign agreement are 92.3–93.8% and 92.6–93.2%, respectively.
These results show high repeatability and cross-judge consistency on the examined samples, with most advantage signs preserved despite differences in individual judgments. The effect of replacing the process judge during actual training is examined separately in A6.
Q6:
How sensitive is the method to inconsistencies or errors in the LLM judge used to estimate accepted support increments?
A6: Replacing the process judge with GLM-5.3 for the latter 100 training steps yields an average score of 61.5, compared with 61.7 for the original Dr.Credit and 58.0 for GRPO.
We resumed Dr.Credit from its step-100 checkpoint, replaced the original Qwen3.6-35B-A3B process judge with GLM-5.3, and continued training to step 200. We compare this run, denoted GLM-resume, with the original 200-step runs below. GLM-5.3 exhibits the lowest credit-level agreement and process-advantage cosine similarity among the judges evaluated in A5, making this a useful test of whether the observed judgment differences affect training outcomes.
| Method | DeepRubric-Val | ResearchQA | DRB | RR | Avg. |
|---|---|---|---|---|---|
| GRPO | 66.9 | 75.4 | 42.5 | 47.1 | 58.0 |
| Dr.Credit | 70.6 | 79.8 | 46.1 | 50.3 | 61.7 |
| GLM-resume | 70.0 | 80.0 | 45.8 | 50.2 | 61.5 |
GLM-resume is 0.2 pp below the original Dr.Credit on average, with a maximum absolute difference of 0.6 pp on any benchmark. It remains 3.5 pp above GRPO on average, with gains on all four benchmarks. Thus, the tested judge replacement preserves most of Dr.Credit's performance advantage despite the scoring differences observed in A5. Together with the agreement analysis, this provides evidence of robustness to the tested process-judge variation during continued training.
Q7:
Could judge disagreement or uncertainty be incorporated into the credit assignment mechanism to reduce noisy supervision?
A7: Yes. Judge disagreement or uncertainty could provide an additional signal for refining credit assignment within Dr.Credit. Although the current mechanism does not explicitly weight credit by uncertainty, a possible extension would use agreement across repeated assessments or different judges to adjust the credit assigned to an incremental contribution. The same signal could guide evidence acceptance: disputed support points could remain pending reassessment before entering the accepted history. This would keep credit assignment and history updates aligned, while allowing uncertain contributions to be reconsidered when further evidence becomes available.
As shown in A6, Dr.Credit retains its performance advantage over GRPO under the tested process-judge replacement, providing evidence of robustness in this setting. Building on these results, the suggested uncertainty-aware extension offers a promising direction for further reducing supervision noise and improving robustness.
Response to Reviewer edtQ (methodological novelty)
Q1:
A1: Dr.Credit uses the task’s final-report rubrics as a shared reference for assigning credit to intermediate research steps. The key methodological difference is what the supervision measures: existing rubric-based outcome rewards assess how well a completed response satisfies the requirements, whereas our process credit assesses the additional evidence support that a research step provides toward meeting those requirements.
This extension requires more than applying the final-report scoring rule to individual steps. A report-level rubric specifies what the completed report should address, but a research step typically supplies only part of the evidence needed to fulfill it. Such a step can be useful without independently satisfying the entire requirement. Moreover, relevance alone does not establish contribution: evidence may support a rubric while merely repeating information already acquired.
For example, suppose a rubric requires comparing the effectiveness and applicability of two techniques. A turn that retrieves effectiveness data for one technique provides partial but useful support; a later turn that adds a previously missing applicability condition extends that support. Another turn that simply repeats the same data adds no further support. All three observations may be relevant to the rubric, but their contributions depend on what the trajectory has already established.
Dr.Credit makes this distinction explicit by maintaining an accepted support history for each rubric and assessing newly acquired evidence against that history. The rubric determines which evidence supports the final task requirements; the history provides the reference for distinguishing new support, meaningful extensions, and repetition. The resulting assessments supply turn-level process credit without requiring a canonical final answer, additional step-level human annotations, or separately generated process rubrics.
RaR, DeepRubric, and DR Tulu use rubrics to construct outcome rewards for completed responses. Dr.Credit builds on this evaluation framework by defining how intermediate evidence contributes to the same requirements and using those contributions to supervise research steps.
Q2:
A2: The key difference is the reference used to define intermediate progress. InfoReasoner studies question answering with ground-truth answers. Its Eq. (17) and Algorithm 1 explicitly use the ground-truth answer to identify the correct semantic class, then reward an increase in that class’s log probability after retrieval. Eq. (15) also describes semantic-entropy reduction without accessing the answer label, but this measures whether the model’s answer distribution becomes more concentrated; it does not establish whether an open-ended report better satisfies its evaluation requirements.
For open-ended reports, satisfying an evaluation requirement need not mean converging toward a predetermined answer. For example, a requirement to analyze a proposal’s applicability and limitations can be satisfied through different evidence and analyses that are not semantically equivalent. New evidence may uncover an additional limitation and make the conclusion more qualified while improving the report. Thus, concentration around an answer class is not a direct measure of progress toward satisfying such a requirement. The difficulty is defining an appropriate target answer, rather than simply estimating its probability.
Dr.Credit instead uses the final-report rubric set as the reference for intermediate credit. It assesses whether newly acquired evidence adds or extends support for each rubric relative to previously accepted support. This grounds process supervision in the evidence accumulated toward the report’s requirements, without requiring a canonical target report or treating convergence toward an answer class as the measure of research progress.
Q3:
A3: The novelty lies primarily in formulating process credit as incremental evidence support for the final report’s requirements. Dr.Credit directly reuses the task-specific rubric set for final-report evaluation to supervise intermediate research steps, enabling process credit assignment in open-ended tasks without a canonical answer or separately generated process rubrics.