acceptodds
Desk rejected at ICLR 2027

Dr.Credit: Rubric-Grounded Process Credit Assignment for Deep Research Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Rubric-based tasks are increasingly addressed through reinforcement learning (RL), with rubric scores used as training rewards. However, these rewards typically supervise final answers without distinguishing the contributions of intermediate decisions. Many existing credit assignment methods rely on ground-truth answers to define process rewards, limiting their applicability to open-ended tasks without canonical solutions. To address this limitation, the proposed rubric-grounded credit uses task requirements as a shared reference for final answer evaluation and process supervision. The information returned by tools is assessed for the additional support it provides toward satisfying each rubric relative to that rubric's history of accepted support. By referencing these histories, credit distinguishes new support from evidence already present in the trajectory while recognizing partial support for each rubric. Dr.Credit uses rubric-grounded credit to supervise intermediate tool turns in an RL framework for deep research agents. The resulting process advantages are combined with GRPO outcome advantages to guide research decisions while retaining supervision of final-report quality. Evaluations on four in-domain and out-of-domain benchmarks show that Dr.Credit outperforms the evaluated open deep research baselines on every primary metric and submetric. Meanwhile, with an 8B-parameter backbone, the trained agent achieves average performance competitive with the evaluated frontier proprietary models. Further analyses suggest more efficient evidence acquisition and higher-quality reports under limited research-turn budgets, motivating the extension of rubric-grounded process supervision to a broader range of rubric-based tasks.

Primary area:
reinforcement learning
Keywords:
DeepResearch, Rubrics as Rewards, Credit Assignment, Agentic RL

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_edtQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the challenge of providing turn-level supervision for reinforcement learning (RL) of deep-research agents on open-ended tasks without canonical answers. The proposed approach uses task-specific requirements as a shared reference for both final-answer evaluation and intermediate process supervision. In particular, it introduces a credit assignment mechanism that measures the incremental support contributed by each research step towards satisfying these requirements. The method is evaluated on four benchmarks, including DeepRubric as the in-domain benchmark and three others for out-of-domain generalisation.

After rebuttal: The authors responses partially address my concerns in review, and I have updated my overall score accordingly.

Critical strengths

S1. Comprehensive baseline comparisons. The evaluation covers a broad range of baselines, including proprietary frontier models, open-source models, conventional RAG approaches, and 8B variants of both open-search and deep-research agents. This provides a useful basis for assessing the proposed method against different approaches to research-agent design and training.

S2. Useful ablation studies. The ablation studies in Tables 2 and 3 help examine the contributions of different components of the proposed approach, providing additional evidence beyond the overall performance comparisons.

Critical weaknesses

W1. Insufficiently differentiated novelty. The paper should more clearly distinguish its contributions from existing work. Rubric-based rewards have already been explored in RaR, DeepRubric, and DR Tulu, including applications to deep-research agents. Similarly, using intermediate signals to guide RL training for research agents is not new, as demonstrated by InfoReasoner. The key question is therefore whether the proposed requirement-grounded incremental credit assignment introduces a fundamentally different supervision signal or mainly combines existing ideas. A more explicit conceptual and empirical comparison with these closely related approaches is needed.

W2. Reliability and computational cost of credit assignment. The proposed method relies on LLM judges to estimate accepted support increments (Equation 4), making the quality of turn-level supervision dependent on judge reliability. The paper provides insufficient analysis of judge consistency, potential evaluation biases, and how erroneous intermediate judgments may affect RL training. In addition, the method introduces substantial computational overhead. Table 7 reports a latency exceeding 23 seconds, compared with approximately 2.4 seconds for GRPO, representing nearly a tenfold increase. This raises concerns about its practical scalability and whether the performance improvements justify the additional training cost.

Questions for authors

Q1. Novelty relative to prior work: What is the key methodological difference between the proposed requirement-grounded credit assignment and existing rubric-based reward approaches such as RaR, DeepRubric, and DR Tulu? How does it differ from intermediate reward mechanisms such as InfoReasoner? In particular, is the novelty primarily in the reward formulation, the use of task requirements, or the estimation of incremental support?

Q2. Judge reliability: How sensitive is the method to inconsistencies or errors in the LLM judge used to estimate accepted support increments? Have the authors measured agreement across different judges, repeated evaluations, or human annotations? Could judge disagreement or uncertainty be incorporated into the credit assignment mechanism to reduce noisy supervision?

Q3. Computational efficiency: Given the nearly tenfold latency increase over GRPO reported in Table 7, how does the method compare under matched computational budgets? Could the authors report performance against total training time, GPU-hours, or judge calls, and investigate whether less frequent or selective intermediate evaluations could achieve similar gains at lower cost?

6 comments
AuthorsOpenReview ↗

Response to Reviewer edtQ (methodological novelty)

Q1:

Novelty relative to prior work: What is the key methodological difference between the proposed requirement-grounded credit assignment and existing rubric-based reward approaches such as RaR, DeepRubric, and DR Tulu?

A1: Dr.Credit uses the task’s final-report rubrics as a shared reference for assigning credit to intermediate research steps. The key methodological difference is what the supervision measures: existing rubric-based outcome rewards assess how well a completed response satisfies the requirements, whereas our process credit assesses the additional evidence support that a research step provides toward meeting those requirements.

This extension requires more than applying the final-report scoring rule to individual steps. A report-level rubric specifies what the completed report should address, but a research step typically supplies only part of the evidence needed to fulfill it. Such a step can be useful without independently satisfying the entire requirement. Moreover, relevance alone does not establish contribution: evidence may support a rubric while merely repeating information already acquired.

For example, suppose a rubric requires comparing the effectiveness and applicability of two techniques. A turn that retrieves effectiveness data for one technique provides partial but useful support; a later turn that adds a previously missing applicability condition extends that support. Another turn that simply repeats the same data adds no further support. All three observations may be relevant to the rubric, but their contributions depend on what the trajectory has already established.

Dr.Credit makes this distinction explicit by maintaining an accepted support history for each rubric and assessing newly acquired evidence against that history. The rubric determines which evidence supports the final task requirements; the history provides the reference for distinguishing new support, meaningful extensions, and repetition. The resulting assessments supply turn-level process credit without requiring a canonical final answer, additional step-level human annotations, or separately generated process rubrics.

RaR, DeepRubric, and DR Tulu use rubrics to construct outcome rewards for completed responses. Dr.Credit builds on this evaluation framework by defining how intermediate evidence contributes to the same requirements and using those contributions to supervise research steps.

Q2:

How does it differ from intermediate reward mechanisms such as InfoReasoner?

A2: The key difference is the reference used to define intermediate progress. InfoReasoner studies question answering with ground-truth answers. Its Eq. (17) and Algorithm 1 explicitly use the ground-truth answer to identify the correct semantic class, then reward an increase in that class’s log probability after retrieval. Eq. (15) also describes semantic-entropy reduction without accessing the answer label, but this measures whether the model’s answer distribution becomes more concentrated; it does not establish whether an open-ended report better satisfies its evaluation requirements.

For open-ended reports, satisfying an evaluation requirement need not mean converging toward a predetermined answer. For example, a requirement to analyze a proposal’s applicability and limitations can be satisfied through different evidence and analyses that are not semantically equivalent. New evidence may uncover an additional limitation and make the conclusion more qualified while improving the report. Thus, concentration around an answer class is not a direct measure of progress toward satisfying such a requirement. The difficulty is defining an appropriate target answer, rather than simply estimating its probability.

Dr.Credit instead uses the final-report rubric set as the reference for intermediate credit. It assesses whether newly acquired evidence adds or extends support for each rubric relative to previously accepted support. This grounds process supervision in the evidence accumulated toward the report’s requirements, without requiring a canonical target report or treating convergence toward an answer class as the measure of research progress.

Q3:

In particular, is the novelty primarily in the reward formulation, the use of task requirements, or the estimation of incremental support?

A3: The novelty lies primarily in formulating process credit as incremental evidence support for the final report’s requirements. Dr.Credit directly reuses the task-specific rubric set for final-report evaluation to supervise intermediate research steps, enabling process credit assignment in open-ended tasks without a canonical answer or separately generated process rubrics.

AuthorsOpenReview ↗

Response to Reviewer edtQ (comparison with prior work)

Q4:

A more explicit conceptual and empirical comparison with these closely related approaches is needed.

A4: The following table summarizes the conceptual differences in supervision source and the quantity being rewarded.

MethodOpen-ended generationProcess rewardsKey distinction
Mathematical PRMs: Let's Verify Step by Step; Math-ShepherdNo: mathematical reasoningYesTrain PRMs using human step labels or automatically derived labels based on reference answers, respectively.
RaRYes: includes open-ended medical responsesNoUses task-specific rubrics to reward final responses.
DeepRubricYesNoConstructs evidence-tree-grounded rubrics for final-report rewards.
DR TuluYesNoEvolves final-report evaluation rubrics using retrieved evidence and on-policy responses.
InfoReasonerNo: QA with reference answersYesScores retrieval through changes in the probability of the correct-answer class (Eq. 17), then aggregates these scores into a trajectory reward.
RubricEMYesYesCompares trajectories for the same question to generate and evolve stage-specific scoring rubrics for stage-level rewards.

Here, “open-ended generation” refers to tasks whose output quality cannot be determined solely by a single reference answer; “process rewards” assess the quality or contribution of individual steps or stages. The InfoReasoner entry describes its Eq. (17); its reference-free semantic-entropy formulation is discussed in A2.

Dr.Credit directly reuses final-report rubrics to assess intermediate evidence contributions, with per-rubric support histories for Visit. This provides process supervision for open-ended tasks without a canonical answer or separately generated process rubrics.

AuthorsOpenReview ↗

Response to Reviewer edtQ (process-judge reliability and training sensitivity)

Q5:

Have the authors measured agreement across different judges, repeated evaluations, or human annotations?

A5: Yes. We conducted same-model repeated evaluation and cross-judge evaluation on 256 training query groups, comparing the resulting credit levels and advantages with those from the original training assessments. Qwen3.6-35B-A3B is the original process judge; its row measures repeatability, while DeepSeek-Flash and GLM-5.3 measure cross-model agreement.

We assess both the judgments and the training signals derived from them:

  • Level agreement: the percentage of re-evaluated credit levels matching the original levels.
  • Process cosine: cosine similarity between the original process advantages and those recomputed from the re-evaluated credits using the same group normalization.
  • Process sign agreement: the percentage of process advantages whose signs agree before and after re-evaluation.
  • Fused sign agreement: sign agreement after combining the recomputed process advantages with the unchanged original outcome advantages.
Re-evaluation judgeLevel agreement (%)Process cosineProcess sign agreement (%)Fused sign agreement (%)
Qwen3.6-35B-A3B95.40.96897.497.1
DeepSeek-Flash93.50.94393.893.2
GLM-5.389.70.91292.392.6

Same-model re-evaluation yields 95.4% level agreement and 0.968 process-advantage cosine similarity. Across the two alternative judges, level agreement is 89.7–93.5%, process-advantage cosine similarity is 0.912–0.943, and process and fused sign agreement are 92.3–93.8% and 92.6–93.2%, respectively.

These results show high repeatability and cross-judge consistency on the examined samples, with most advantage signs preserved despite differences in individual judgments. The effect of replacing the process judge during actual training is examined separately in A6.

Q6:

How sensitive is the method to inconsistencies or errors in the LLM judge used to estimate accepted support increments?

A6: Replacing the process judge with GLM-5.3 for the latter 100 training steps yields an average score of 61.5, compared with 61.7 for the original Dr.Credit and 58.0 for GRPO.

We resumed Dr.Credit from its step-100 checkpoint, replaced the original Qwen3.6-35B-A3B process judge with GLM-5.3, and continued training to step 200. We compare this run, denoted GLM-resume, with the original 200-step runs below. GLM-5.3 exhibits the lowest credit-level agreement and process-advantage cosine similarity among the judges evaluated in A5, making this a useful test of whether the observed judgment differences affect training outcomes.

MethodDeepRubric-ValResearchQADRBRRAvg.
GRPO66.975.442.547.158.0
Dr.Credit70.679.846.150.361.7
GLM-resume70.080.045.850.261.5

GLM-resume is 0.2 pp below the original Dr.Credit on average, with a maximum absolute difference of 0.6 pp on any benchmark. It remains 3.5 pp above GRPO on average, with gains on all four benchmarks. Thus, the tested judge replacement preserves most of Dr.Credit's performance advantage despite the scoring differences observed in A5. Together with the agreement analysis, this provides evidence of robustness to the tested process-judge variation during continued training.

Q7:

Could judge disagreement or uncertainty be incorporated into the credit assignment mechanism to reduce noisy supervision?

A7: Yes. Judge disagreement or uncertainty could provide an additional signal for refining credit assignment within Dr.Credit. Although the current mechanism does not explicitly weight credit by uncertainty, a possible extension would use agreement across repeated assessments or different judges to adjust the credit assigned to an incremental contribution. The same signal could guide evidence acceptance: disputed support points could remain pending reassessment before entering the accepted history. This would keep credit assignment and history updates aligned, while allowing uncertain contributions to be reconsidered when further evidence becomes available.

As shown in A6, Dr.Credit retains its performance advantage over GRPO under the tested process-judge replacement, providing evidence of robustness in this setting. Building on these results, the suggested uncertainty-aware extension offers a promising direction for further reducing supervision noise and improving robustness.

Reviewer_ZUgbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a rubric-grounded process credit assignment method for training deep research agents. The method assigns turn-level rewards to search and visit actions based on how much new evidence they contribute towards each rubric, while maintaining evidence histories to avoid rewarding repeated information. The process rewards are combined with GRPO outcome rewards during training. Experiments across 4 benchmarks show consistent improvements over existing open deep research baselines.

Critical strengths
  1. I think this is a timely and interesting problem. Rubric-based process supervision for open-ended tasks is still a relatively new direction, with plenty of room for further research.

  2. The empirical results are quite strong. The authors conducted a fairly extensive set of experiments, including comparisons across multiple benchmarks, ablation studies on different components, and analyses of computational efficiency.

Critical weaknesses
  1. My main concern is the novelty compared to existing work. For example, RubricEM (https://arxiv.org/abs/2605.10899) has already explored rubric-based process supervision for deep research agents, using stage-level rewards. While this work introduces turn-level rewards and history-aware evidence tracking, I'm not entirely sure how substantial this difference is. The paper compares against other methods like Dr. Tulu and DeepRubric, but these primarily use outcome-level rubric rewards. I think a more direct comparison with baselines like RubricEM, or at least a simpler rubric-based process reward baseline, would help better establish the contribution.

  2. The paper motivates the approach by arguing that "existing credit assignment methods rely on ground truth answers to define process rewards, limiting their applicability to open-ended tasks without canonical solutions", but rubric-based methods still heavily rely on high-quality, task-specific rubrics, which are another form of supervision. In this work, the authors directly use the existing DeepRubric dataset rather than addressing how these rubrics can be constructed or adapted. I'm not entirely convinced this assumption generalizes well to real-world open-ended questions, where the expectations for a good answer may vary depending on the audience or context.

  3. Although the authors report lower overall wall-clock training time, process judging introduces substantial additional computation (almost a 10x increase per trajectory). I'm not sure whether the finer-grained supervision justifies this additional cost, especially compared to approaches like RubricEM that use stage-level supervision. It would be helpful to better understand the trade-off between performance gains and the total computational cost, including the additional judging resources.

Questions for authors

Have the authors considered evaluating the robustness of the method to incomplete or imperfect rubrics? For example, how would performance change if some rubric criteria were removed or replaced with less comprehensive rubrics?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ZUgb (novelty and process-reward baseline)

Q1:

My main concern is the novelty compared to existing work. For example, RubricEM (https://arxiv.org/abs/2605.10899) has already explored rubric-based process supervision for deep research agents, using stage-level rewards. While this work introduces turn-level rewards and history-aware evidence tracking, I'm not entirely sure how substantial this difference is. The paper compares against other methods like Dr. Tulu and DeepRubric, but these primarily use outcome-level rubric rewards. I think a more direct comparison with baselines like RubricEM, or at least a simpler rubric-based process reward baseline, would help better establish the contribution.

A1: Dr.Credit's central contribution is to reuse final-report rubrics to assess the incremental contributions of intermediate research steps. We have added a RubricEM-Style process-reward baseline, which achieves an average score of 59.2, compared with 58.0 for GRPO and 61.7 for Dr.Credit.

Methodological distinction. We agree that RubricEM is a closer comparison than methods using only outcome-level rubric rewards. RubricEM generates stage-specific scoring rubrics by comparing trajectories for the same question, while retaining the training-data rubrics for the final Answer stage (Appendix C). Dr.Credit instead uses the final-report rubric set itself as the reference for intermediate credit assignment, without generating a separate set of process rubrics.

This reuse defines what progress means: each evidence-acquisition step is assessed by the additional support it contributes toward the final report's requirements. The per-rubric history provides the reference against which that contribution is judged, distinguishing new or extended support from information already obtained. Thus, the difference concerns both the supervision source and the quantity being rewarded, beyond the choice of turn-level versus stage-level granularity.

Direct process-reward comparison. Because official training code and model checkpoints were unavailable for our comparison, we implemented RubricEM-Style from the paper's description. A strong model, DeepSeek-Flash, compares research trajectories for the same question and generates process rubrics, which are then used to construct process rewards. Reward normalization and advantage aggregation follow the same design as Dr.Credit. This is an adaptation of RubricEM's process-rubric generation mechanism within our training framework, rather than a reproduction of its complete framework.

MethodDeepRubric-ValResearchQADRBRRAvg.
GRPO66.975.442.547.158.0
RubricEM-Style67.577.244.447.759.2
Dr.Credit70.679.846.150.361.7

RubricEM-Style improves over GRPO by 1.2 pp on average, supporting the value of rubric-based process supervision. Dr.Credit further improves over this baseline by 2.5 pp, with gains of 3.1, 2.6, 1.7, and 2.6 pp across the four benchmarks. Separately, the submitted history ablation (Table 3b) shows that removing the per-rubric evidence history reduces performance by 3.0 pp on average across the three evaluated benchmarks. Together, these results support the proposed credit design beyond adding a generic rubric-based process reward; they do not isolate reward granularity as the sole cause of the improvement.

AuthorsOpenReview ↗

Response to Reviewer ZUgb (rubric dependence and robustness)

Q2:

The paper motivates the approach by arguing that "existing credit assignment methods rely on ground truth answers to define process rewards, limiting their applicability to open-ended tasks without canonical solutions", but rubric-based methods still heavily rely on high-quality, task-specific rubrics, which are another form of supervision. In this work, the authors directly use the existing DeepRubric dataset rather than addressing how these rubrics can be constructed or adapted. I'm not entirely convinced this assumption generalizes well to real-world open-ended questions, where the expectations for a good answer may vary depending on the audience or context.

Have the authors considered evaluating the robustness of the method to incomplete or imperfect rubrics? For example, how would performance change if some rubric criteria were removed or replaced with less comprehensive rubrics?

A2: Rubrics are indeed a form of supervision, and Dr.Credit does not remove the need to specify what constitutes a good answer. Our contribution is to derive process supervision from those outcome requirements. To examine dependence on the original DeepRubric rubrics, we retrained both methods with online-generated rubrics and with randomly incomplete rubric sets. Dr.Credit retains an advantage over GRPO in both settings.

Online-generated rubrics (RLER). We use a DR Tulu-style online rubric-generation procedure, with DeepSeek-Flash as the strong model generating the rubrics. Both GRPO and Dr.Credit are trained under this alternative rubric construction. This tests whether Dr.Credit's gains depend on using the original DeepRubric rubric sets.

Incomplete rubrics (20%). For each training query, we randomly delete 20% of its rubrics, rounding the number to the nearest integer, and retrain both methods. With approximately 5–9 rubrics per query, this removes 1–2 rubrics. This directly tests the effect of omitting some evaluation requirements during training.

MethodDeepRubric-ValResearchQADRBRRAvg.
GRPO66.975.442.547.158.0
Dr.Credit70.679.846.150.361.7
GRPO (RLER)63.874.042.944.256.2
Dr.Credit (RLER)68.578.745.648.960.4
GRPO (20%)65.375.942.845.657.4
Dr.Credit (20%)69.878.546.350.061.2

With online-generated rubrics, Dr.Credit achieves 60.4 versus 56.2, a 4.2 pp average advantage. Its score decreases by 1.3 pp relative to the original rubrics, compared with 1.8 pp for GRPO. With 20% deletion, Dr.Credit achieves 61.2 versus 57.4, a 3.8 pp advantage; its average decreases by 0.5 pp, compared with 0.6 pp for GRPO. Dr.Credit outperforms the corresponding GRPO baseline on every benchmark in both settings.

Changing rubrics affects final performance because they define the reward objective—a dependence inherent to the RaR paradigm. Dr.Credit shows smaller average performance drops than GRPO under both online rubric replacement (1.3 versus 1.8 pp) and 20% deletion (0.5 versus 0.6 pp), while retaining its performance advantage in both settings.

As complementary evidence beyond DeepRubric, we also trained both methods on 6,400 query–rubric pairs from RubricHub Writing, using web_search, web_visit, paper_search, and python_interpreter. Dr.Credit improves over GRPO by 1.1, 3.3, and 2.7 pp on a 512-example held-out writing set, WritingBench, and CreateWriting-V3, respectively. This supports using the same process-credit principle with another domain's outcome rubrics, while retaining the requirement for task-appropriate supervision.

AuthorsOpenReview ↗

Response to Reviewer ZUgb (total training cost and performance)

Q3:

Although the authors report lower overall wall-clock training time, process judging introduces substantial additional computation (almost a 10x increase per trajectory). I'm not sure whether the finer-grained supervision justifies this additional cost, especially compared to approaches like RubricEM that use stage-level supervision. It would be helpful to better understand the trade-off between performance gains and the total computational cost, including the additional judging resources.

A3: Under the same allocation of 16 training PPUs and 64 external-service PPUs, Dr.Credit uses 14.9% fewer allocated PPU-hours than GRPO and 35.7% fewer than RubricEM-Style over 200 training steps, while achieving the highest average score: 61.7 versus 58.0 and 59.2, respectively.

Although per-trajectory scoring latency increases from 2.40 to 23.03 seconds, this alone does not determine total training cost. End-to-end runtime depends on the scheduling of rollout generation, external-service inference, and policy updates across the training system. We therefore compare the measured runtime and total allocated PPU-hours of complete training runs, including scoring.

Full resource accounting. All three methods use 16 training PPUs plus 64 external-service PPUs. For GRPO and Dr.Credit, the external pool hosts 64 Qwen3.6-35B-A3B instances that summarize retrieved webpage content for the web-visit tool and evaluate final reports. Dr.Credit also performs process judging on this same pool, without adding devices.

Resource, 200 training stepsGRPODr.CreditRubricEM-Style
Training / external-service PPUs16 / 6416 / 6416 / 64
Recorded training time, including scoring (hours)75.1463.9699.48
Training PPU-hours1,202.191,023.321,591.68
External-service PPU-hours4,808.774,093.276,366.72
Total PPU-hours6,010.975,116.587,958.40
Four-benchmark average score58.061.759.2

PPU-hours equal allocated device count multiplied by recorded training time. The table excludes RubricEM-Style's additional DeepSeek-Flash fees for process-rubric generation; neither GRPO nor Dr.Credit incurs these fees.

Dr.Credit improves the average score by 3.7 pp over GRPO and 2.5 pp over RubricEM-Style, while requiring fewer allocated device-hours than either baseline. Despite the longer per-trajectory scoring latency, Dr.Credit completes training sooner on the same hardware, achieving higher performance with fewer total PPU-hours.

Performance at a closely matched resource budget. Taking Dr.Credit's 200-step allocation as the reference, we compare it with GRPO at 171 steps:

MethodTraining stepsTotal PPU-hoursDeepRubric-ValResearchQADRBRRAvg.
GRPO1715,122.7265.872.241.943.755.9
Dr.Credit2005,116.5870.679.846.150.361.7

Dr.Credit uses 0.12% fewer PPU-hours and improves performance on all four benchmarks, with a 5.8 pp average gain. Together, these results show that Dr.Credit’s finer-grained supervision improves the overall cost–performance trade-off: it achieves higher report quality with fewer total PPU-hours than both GRPO and RubricEM-Style, and retains a substantial advantage over GRPO at a closely matched resource budget.

Reviewer_vLyLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose Dr.Credit which essentially adds rubric-based process rewards in addition to the standard outcome based GRPO for deep research agents. An LLM judge is used to score at each turn in a ReACt loop where an auxiliary reward term is added based on additional evidence each turn provides. The method improves over outcome-only GRPO across four research benchmarks.

Critical strengths
  • The method shows gains over the simple GRPO baseline with matched training settings.

  • The idea behind scoring each turn based on how much help it provides to the final rubrics is sensible and the paper has good ablations.

Critical weaknesses
  • The claim of a general approach for open-ended agent RL feels broader than the evidence supports. All experiments involve deep research with just 2 tools -- Search and Visit. Much of the framing and the writing in the paper such as in the Intro section is generic, while the actual reward design is closely tied to only web evidence collection. It is unclear how this transfers to other tasks or agents with more tools. Agents these days use more than those 2 tools for any task.

  • I have a concern about redundant web searches. Agents make similar web searches for multiple turns for any web based task. While "Visit" scoring uses evidence history, snippet scoring is independent across turns. Similar searches could therefore repeatedly earn credit for essentially the same information. The history ablation does not directly address this.

  • There is not much novelty. The main idea is to apply rubric-based LLM judging at intermediate turns and combine those scores with outcome advantages. The conceptual advance over existing process supervision needs clearer justification. GRPO and Process reward models for each step already exist in mathematical reasoning for a few years.

  • Figure 4(a) raises a scaling question. GRPO slightly overtakes Dr.Credit in cumulative evidence at 12–14 turns, while Dr.Credit appears to plateau. This leaves unclear whether its advantage persists in settings like longer horizon tasks, which deep research agents are typically trained/used for.

Questions for authors
  • Can repeated or paraphrased searches receive duplicate snippet credit? What happens when several searches return the same URL that is later visited?

  • What would the reward design look like for a task beyond web research, especially with a larger tool set?

  • Does Dr.Credit keep improving beyond the tested turn budgets if you train agents for longer?

10 comments
AuthorsOpenReview ↗

Response to Reviewer vLyL (novelty)

Q1:

There is not much novelty. The main idea is to apply rubric-based LLM judging at intermediate turns and combine those scores with outcome advantages. The conceptual advance over existing process supervision needs clearer justification. GRPO and Process reward models for each step already exist in mathematical reasoning for a few years.

A1: Dr.Credit's core contribution is to extend process supervision beyond ground-truth-dependent tasks, such as mathematical reasoning, to a broader class of rubric-based open-ended tasks. We design a framework that directly reuses existing task-specific rubrics for final-report evaluation to assess the contributions of intermediate research steps, thereby turning outcome evaluation into process supervision. This enables intermediate steps to be evaluated against the final report's requirements without additional step-level human annotations or process-specific rubric generation, while maintaining alignment between process supervision and final-task objectives.

The table below distinguishes prior work by task scope, process rewards, and supervision source. Here, “open-ended generation” refers to tasks whose output quality cannot be determined solely by a single reference answer; “process rewards” score the quality or contribution of individual steps or stages.

MethodOpen-ended generationProcess rewardsKey distinction
Mathematical PRMs: Let's Verify Step by Step; Math-ShepherdNo: mathematical reasoningYesTrain PRMs using human step labels or automatically derived labels based on reference answers, respectively.
RaRYes: includes open-ended medical responsesNoUses task-specific rubrics to reward final responses.
DeepRubricYesNoConstructs evidence-tree-grounded rubrics for final-report rewards.
DR TuluYesNoEvolves final-report evaluation rubrics using retrieved evidence and on-policy responses.
InfoReasonerNo: QA with reference answersYesScores retrieval through changes in the probability of the correct-answer class (Eq. 17), then aggregates these scores into a trajectory reward.
RubricEMYesYesCompares trajectories for the same question to generate and evolve stage-specific scoring rubrics for stage-level rewards.
Dr.CreditYesYesDirectly reuses final-report rubrics to assess intermediate steps' contributions toward those requirements.

The supervision used by mathematical PRMs and InfoReasoner does not transfer directly to open-ended, long-form report generation. Human process supervision requires additional annotations for research trajectories, while automatic labels or probability gains grounded in reference answers require a verifiable target answer. RaR, DeepRubric, and DR Tulu establish how rubrics can evaluate open-ended responses, but their rubric rewards assess final outputs. Dr.Credit extends these existing evaluation requirements to supervision of intermediate research steps.

Among these approaches, RubricEM is the closest to ours, but the two methods differ in how they obtain rubrics for process supervision. RubricEM retains the existing training-data rubrics for the final Answer stage and separately generates stage-specific scoring rubrics by comparing trajectories for the same question. In contrast, Dr.Credit directly reuses the existing report-level rubrics to supervise intermediate research steps, avoiding additional rubric generation and potential inconsistencies between process-specific criteria and final-report requirements.

Beyond rubric reuse, Dr.Credit accounts for how substantive evidence accumulates over time in primary evidence acquisition. The contribution of tools serving this role depends on the evidence already established: their outputs may introduce new support, meaningfully extend existing support, or merely repeat known information. Dr.Credit therefore maintains a rubric-specific support history and applies history-conditioned assessment to these evidence-acquisition steps, evaluating each contribution against the evidence accumulated beforehand. Unlike RubricEM, which assigns a shared advantage to all policy tokens within a stage, Dr.Credit distinguishes the contributions of individual evidence-acquisition steps, enabling more fine-grained, turn-level process credit assignment.

In the submitted main comparison, our Qwen3-8B agent improves over matched GRPO by 3.5 pp on average across four deep-research benchmarks. Its average score of 61.7 is also competitive with the frontier models evaluated using the same tools, including DeepSeek-V4-Pro (60.4), Opus 4.8 (60.2), and GLM-5.3 (59.8).

AuthorsOpenReview ↗

Response to Reviewer vLyL (repeated snippet credit)

Q2:

I have a concern about redundant web searches. Agents make similar web searches for multiple turns for any web based task. While "Visit" scoring uses evidence history, snippet scoring is independent across turns. Similar searches could therefore repeatedly earn credit for essentially the same information. The history ablation does not directly address this.

A2: Thank you for raising this concern. In our current approach, Search does not perform evidence history verification; this is in fact an engineering simplification motivated by our experimental results. For Search, we compare three approaches: independent scoring (the current Dr.Credit method), first-occurrence scoring (Exact-first), and treating snippets as short observations with history-conditioned assessment (Snippet-as-Observation, which handles Search results in exactly the same way as Visit).

First, we apply Exact-first to fixed trajectories, retaining only the first occurrence of each detected duplicate snippet match. The analysis covers 519 query groups, 4,152 rollouts, 17,851 Search turns, and 32,966 non-final policy turns. Exact-first removes 113 duplicate matches and changes 108 Search credits (0.61%). Relative to the default:

AlternativeProcess RMSEProcess MAEProcess cosineProcess sign agreementFused sign agreement
Exact-first0.01580.00170.999999.98%99.95%

“Process” denotes the normalized process advantage; “Fused” denotes its sum with the outcome advantage. These diagnostics show that removing the detected duplicate matches has little effect on either advantage signal in this sample.

Comparing the final report performance, the current Dr.Credit method without deduplication achieves nearly identical overall results to the strict deduplication approach (Exact-first), with both yielding an average score of 61.7, while Snippet-as-Observation does not perform better either. Therefore, we simplify the scoring pipeline for the Search tool to reduce scoring latency and save resources.

MethodDeepRubric (val)ResearchQADRBRRAvg.
Dr.Credit: Independent70.679.846.150.361.7
Exact-first70.179.446.550.961.7
Snippet-as-Observation70.578.245.049.760.9

We hypothesize that this difference is related to the information characteristics of Search. Unlike the detailed webpage content returned by Visit, Search snippets are brief and fragmented: they may expose only part of a fact, while different results may provide complementary aspects of the same evidence. Applying Visit-style history conditioning would therefore require maintaining and comparing numerous partial support points, increasing scoring complexity without necessarily yielding more reliable credit assignments. This suggests a general principle: in our method, tools can be broadly divided into two categories: fine-grained, history-conditioned credit assignment is used for primary evidence acquisition, while a lighter-weight scoring mechanism can be adopted for auxiliary source discovery.

AuthorsOpenReview ↗

Response to Reviewer vLyL (navigation credit attribution)

Q3:

Can repeated or paraphrased searches receive duplicate snippet credit? What happens when several searches return the same URL that is later visited?

A3: Yes. Repeated or paraphrased searches can receive snippet credit more than once. Similar queries do not necessarily imply redundant information: they may return different snippets or serve different purposes as the research focus evolves. We therefore independently assess whether snippets from results not subsequently visited provide rubric-relevant information at each Search turn, rather than withholding credit based solely on query similarity or occurrence order. A2 further evaluates this design choice by comparing independent scoring with first-occurrence scoring.

When multiple Searches return the same URL and that URL is subsequently visited, the Visit's credit is attributed to every preceding Search that returned it, following Navigation-All. For example, if two Searches return URL u and a later Visit(u) earns 0.5 credit, each Search receives 0.5 navigation credit. The corresponding snippets are excluded from independent snippet scoring. Within each Search, the same Visit is counted only once, and credits from matching Visits are summed and capped at 1.

This attribution reflects the role of Search in source discovery and navigation. A source may appear in an early search but only be selected for a Visit after the research focus becomes clearer. Being the first to return that source establishes temporal priority, but does not imply that later searches played no role in its eventual selection. Importantly, navigation credit is grounded in the subsequent Visit's evidence contribution: a Visit receiving zero credit adds no navigation credit to any preceding Search.

We further compare Navigation-All with Navigation-First, which attributes each Visit's credit only to the earliest Search that returned its URL:

Navigation attributionDeepRubric (val)ResearchQADRBRRAvg.
Navigation-All (Dr.Credit)70.679.846.150.361.7
Navigation-First71.278.845.049.761.2

Navigation-All scores higher on three of the four benchmarks, with an average of 61.7, exceeding Navigation-First by 0.5 pp. Based on this rationale and the experimental results, we retain Navigation-All to supervise all Searches that returned sources subsequently yielding positive evidence contributions.

Reviewer_n1okOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes rubric-grounded process credit for RL on open-ended tasks. An LLM judge scores how much new support each tool turn’s evidence adds for each rubric, compared with a per-rubric history of support already accepted in the rollout. The levels are 0, 0.2 and 0.5. Visit turns are credited directly. Search turns inherit the credit of later visits to URLs they returned, plus 0.1 for each rubric matched by an unvisited snippet. Credits are normalized per group and added to GRPO outcome advantages on non-final turns. A Qwen3-8B agent trained on DeepRubric improves over a matched GRPO baseline by 3.5 points on average across four deep-research benchmarks, and the paper reports performance competitive with frontier models using the same tools.

Critical strengths

GRPO and Dr.Credit share the same SFT starting point, data, outcome reward and training budget. Removing the per-rubric evidence history costs 3.0 points on average (Table 3b), and the fixed-trajectory rescoring in App. B.2 backs this up. Removing the process advantage costs 3.5 points and removing the outcome advantage costs 2.3 (Table 3a).

Critical weaknesses

The training cost is substantially understated. Process supervision adds about 29 judge calls per trajectory (Table 7). That is roughly 1.5M extra calls to a 35B judge over 200 steps (29.09 × 256 trajectories × 200 steps), and scoring latency rises about 10× (23.0 vs 2.4 s per trajectory).

The claims of generality outrun the evidence, and long-horizon scaling is unclear. Contribution (1) presents rubric-grounded credit as a general approach for open-ended agent RL, motivated by writing and medical consultation. However, all experiments are deep research with ≤20 turns.

Questions for authors

What is the total judge compute (tokens or GPU-hours) for Dr.Credit versus GRPO? How do the methods compare at matched total compute? Which judge scores DeepRubric-Val? If it is Qwen3.6-35B-A3B, the in-domain results share a judge with the training reward.

5 comments
AuthorsOpenReview ↗

Response to Reviewer n1ok (training cost)

Q1:

The training cost is substantially understated. Process supervision adds about 29 judge calls per trajectory (Table 7). That is roughly 1.5M extra calls to a 35B judge over 200 steps (29.09 × 256 trajectories × 200 steps), and scoring latency rises about 10× (23.0 vs 2.4 s per trajectory). What is the total judge compute (tokens or GPU-hours) for Dr.Credit versus GRPO?

A1: Including both training devices and the external service pool, Dr.Credit uses 14.9% fewer allocated PPU-hours than GRPO over 200 training steps under the same hardware allocation. Process supervision does introduce additional inference workload: Dr.Credit performs 1,489,244 process-scoring tasks, and per-trajectory scoring latency increases from 2.40 to 23.03 seconds. The end-to-end cost, however, depends on the execution of the full training pipeline.

Both methods use 16 training PPUs and 64 external-service PPUs hosting 64 Qwen3.6-35B-A3B instances. The external pool summarizes retrieved webpage content as part of the web-visit tool and evaluates final reports. Dr.Credit also runs process judging on this same pool, without adding devices. We report device-hours as PPU-hours, reflecting the hardware used in our experiments.

Resource, 200 training stepsGRPODr.Credit
Training / external-service PPUs16 / 6416 / 64
Recorded training time, including scoring (hours)75.1463.96
Training PPU-hours1,202.191,023.32
External-service PPU-hours4,808.774,093.27
Total PPU-hours6,010.975,116.58

PPU-hours are calculated as the allocated device count multiplied by recorded training time, using unrounded times. The external-service row accounts for the entire shared pool, including webpage summarization and all judges.

The complete Dr.Credit run finishes sooner with the same training and service capacity, even after including process scoring. Its total allocation is 5,116.58 versus 6,010.97 PPU-hours, corresponding to 14.9% lower hardware rental cost for the recorded training window at unchanged hourly rates.

AuthorsOpenReview ↗

Response to Reviewer n1ok (matched resource budget)

Q2:

How do the methods compare at matched total compute?

A2: At closely matched total PPU-hour budgets, Dr.Credit achieves an average score of 61.7 versus 55.9 for GRPO, an improvement of 5.8 pp across four benchmarks. We measure the budget using total allocated PPU-hours, including both training and all external services. Both methods use the same 16 training PPUs plus 64 external-service PPUs, covering webpage summarization, report evaluation, and process judging.

Taking Dr.Credit's 200-step allocation as the reference, we compare it with GRPO at 171 steps. Their total allocations are 5,116.58 and 5,122.72 PPU-hours, respectively: Dr.Credit uses 6.14 fewer PPU-hours (0.12% less).

MethodTraining stepsTotal PPU-hoursDeepRubric-ValResearchQADRBRRAvg.
GRPO1715,122.7265.872.241.943.755.9
Dr.Credit2005,116.5870.679.846.150.361.7

Dr.Credit improves scores on all four benchmarks, by 4.8, 7.6, 4.2, and 6.6 pp, respectively. These results support its end-to-end resource efficiency: it achieves higher task performance within a slightly lower total PPU-hour budget, with the external judging infrastructure fully included.

AuthorsOpenReview ↗

Response to Reviewer n1ok (evaluation judges)

Q3:

Which judge scores DeepRubric-Val? If it is Qwen3.6-35B-A3B, the in-domain results share a judge with the training reward.

A3: The original DeepRubric-Val evaluation uses Qwen3.6-35B-A3B, which is also used for the training reward. To examine whether the observed advantage depends on this shared judge, we evaluated both methods with three additional rubric judges: DeepSeek-Flash, GPT-OSS-120B, and GLM-5.3.

MethodQwen3.6-35B-A3BDeepSeek-FlashGPT-OSS-120BGLM-5.3Avg.
GRPO66.967.465.866.166.6
Dr.Credit70.671.267.970.069.9

Dr.Credit scores higher under all three alternative judges, with gains of 3.8, 2.1, and 3.9 pp, respectively, compared with 3.7 pp under the original judge. These results show that its advantage on DeepRubric-Val is also observed when evaluation uses models other than the training judge.

Comments on the paper

1 comment

Dear Reviewers,

As the discussion phase draws to a close, please acknowledge the author's response and let them know whether it has addressed some or all of your concerns.

Thank you for the time and effort you and the other reviewers have put into this discussion.

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.