- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes Dr.Q, a credit assignment method for tool-using agents that combines trajectory-level outcomes, segment-level process rewards, and comparisons across recurring work states. These signals are weighted and combined within a shared PPO update. The paper also examines when local comparisons lose reward contrast and how missing tool feedback can discourage exploration. Validation includes a GRPO comparison on τ-bench, a 200-question 2WikiMultihopQA evaluation, and additional configuration studies, replay analysis, and SQL examples.
Critical strengths
The research question is interesting. Different decisions within an agent trajectory can contribute very differently to its final outcome, yet terminal rewards provide limited guidance about these contributions. Studying how credit granularity, comparison groups, and feedback availability interact is a worthwhile direction. The observation that missing feedback can be mistaken for poor behavior is particularly interesting, as it connects credit assignment to limitations of the training environment.
Critical weaknesses
The experimental validation is substantially insufficient for the scope of the proposed method. The main τ-bench comparison includes only SFT and GRPO, without comparisons against closely related credit assignment methods such as GiGPO or Turn-PPO. On 2WikiMultihopQA, the main 27B evaluation uses only 200 development questions and compares Dr.Q against earlier CPT and SFT checkpoints, rather than a matched GRPO run. This demonstrates improvement after RL, but does not establish the advantage of the proposed estimator. The supplementary experiments provide useful observations, yet do not close these central comparison gaps.
The component studies do not adequately establish generalization or robustness. The 4B ablations mainly report training metrics averaged over the last 25 steps, rather than held-out performance. Moreover, the comparison grouping used for the component-disabled runs is not clearly identified in Table 6, despite grouping itself having a large effect. No repeated-run uncertainty or systematic sensitivity analysis is reported. Given the number of interacting components and weighting choices, these omissions make it difficult to determine whether the gains are reliable and which mechanisms are responsible.
Questions for authors
Do you have matched comparisons against GRPO on the 27B 2Wiki setting and against related credit assignment methods on τ-bench?