acceptodds
Withdrawn from ICLR 2027

Dr.Q: Multi-granularity Credit Decomposition For Long-Horizon Tool-Using Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-horizon tool-using LLM agents fail for reasons that final rewards rarely localize. We observe that such trajectories have a natural recursive structure: a rollout is a sequence of segments, and each segment contains a sequence of states. Credit assignment at each granularity level requires different mechanisms, and—critically—if any granularity level’s local reward is constant within its comparison group, the policy stops decomposing the task at that level. We formalize this as the granularity degeneracy principle and show it is not caught by existing dynamic-sampling filters (e.g., DAPO). We prove that the Anchor’s cross-trajectory comparison is biased when prefixes differ (Proposition 1), and that constant local reward at any granularity causes the policy to stop exploring that granularity (Proposition 2). The latter is a general property of group-relative methods, not specific to DR.Q. On τ-bench, Qwen3.5-9B-DR.Q improves average success from 27.57 to 40.26 over GRPO (5 seeds, ±2.1). Removal ablations show the Rubric contributes +4.8 points, the Anchor +3.2 points, and the Router +3.3 points. We demonstrate granularity degeneracy empirically: in a partially covered environment, the policy stops exploring tools with constant-zero local reward, reducing execution accuracy by 9.4 points. Code is available at https://anonymous.4open.science/r/DRQ-Research-4599.

Primary area:
reinforcement learning
Keywords:
Long-horizon, credit assignment, agentic RL

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cBPzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Dr.Q, a credit assignment method for tool-using agents that combines trajectory-level outcomes, segment-level process rewards, and comparisons across recurring work states. These signals are weighted and combined within a shared PPO update. The paper also examines when local comparisons lose reward contrast and how missing tool feedback can discourage exploration. Validation includes a GRPO comparison on τ-bench, a 200-question 2WikiMultihopQA evaluation, and additional configuration studies, replay analysis, and SQL examples.

Critical strengths

The research question is interesting. Different decisions within an agent trajectory can contribute very differently to its final outcome, yet terminal rewards provide limited guidance about these contributions. Studying how credit granularity, comparison groups, and feedback availability interact is a worthwhile direction. The observation that missing feedback can be mistaken for poor behavior is particularly interesting, as it connects credit assignment to limitations of the training environment.

Critical weaknesses

The experimental validation is substantially insufficient for the scope of the proposed method. The main τ-bench comparison includes only SFT and GRPO, without comparisons against closely related credit assignment methods such as GiGPO or Turn-PPO. On 2WikiMultihopQA, the main 27B evaluation uses only 200 development questions and compares Dr.Q against earlier CPT and SFT checkpoints, rather than a matched GRPO run. This demonstrates improvement after RL, but does not establish the advantage of the proposed estimator. The supplementary experiments provide useful observations, yet do not close these central comparison gaps.

The component studies do not adequately establish generalization or robustness. The 4B ablations mainly report training metrics averaged over the last 25 steps, rather than held-out performance. Moreover, the comparison grouping used for the component-disabled runs is not clearly identified in Table 6, despite grouping itself having a large effect. No repeated-run uncertainty or systematic sensitivity analysis is reported. Given the number of interacting components and weighting choices, these omissions make it difficult to determine whether the gains are reliable and which mechanisms are responsible.

Questions for authors

Do you have matched comparisons against GRPO on the 27B 2Wiki setting and against related credit assignment methods on τ-bench?

Reviewer_mVZGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Dr.Q addresses credit assignment in tool-using agents by combining trajectory-level outcomes, segment-level process rewards, and outcome comparisons across recurring work states within PPO updates. It also examines how reward variation and missing tool feedback affect learning. The reported 9B τ-bench success rate improves from 27.57% with GRPO to 40.26%, while 27B answer EM on a 200-question 2WikiMultihopQA development subset increases from 62.5% after SFT to 69.0% after Dr.Q training.

Critical strengths
  1. The framework separates task completion, process quality, and recurring-state comparisons, with explicit grouping rules and mechanisms for handling repeated visits and sparse anchor groups.
  2. Improvements across both τ-bench domains. Dr.Q outperforms GRPO by 17.39 percentage points in retail and 8.00 points in airline, providing empirical support in both evaluated domains.
Critical weaknesses
  1. Unclear contribution of the proposed credit assignment method. Compared with outcome-only GRPO, Dr.Q introduces process rewards and several additional training designs, but lacks controlled comparisons under matched process supervision and training budgets. Direct comparisons with relevant methods discussed in the paper, such as GiGPO, SALT, and Turn-PPO, are also missing. In particular, the 27B 2Wiki experiment compares only CPT, SFT, and Dr.Q, demonstrating improvement over SFT without establishing the advantage over standard GRPO. (§6, Tables 1–2)

  2. Unclear configurations in the component ablations. Table 6 includes ablations of Gate, Anchor, segment return, and position weighting, but does not specify whether these runs use task grouping or route grouping. All results reported in Table 6 are training metrics, so the final evaluation performance of these configurations remains unclear.

  3. Limited evidence from held-out evaluation and long-horizon tasks. The main 4B configuration comparisons report statistics from the final 25 training steps, while the 27B 2Wiki evaluation covers only 200 questions. Although a 1,533-question BIRD evaluation was completed, the paper mainly presents five selected success cases.

  4. Unusual presentation. The second paragraph of the introduction explicitly states that some figures and tables still retain the previous method names without being updated. The statement “GSM8K exact-match accuracy (%) for the earlier direct-generation variant” indicates that the reported checkpoint is not the latest one. Moreover, the gains for the 9B and 27B models are very small. Figure 4(b) lacks a legend.

Questions for authors

See the weakness.

Reviewer_1y9qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Dr.Q, a credit assignment method for tool-using language agents. It combines three signals: relative trajectory outcomes, discounted process rewards centered by task and segment position, and terminal-outcome comparisons across rollouts sharing a work-state descriptor. These signals are combined in a PPO-style update. The experiments report improvements over GRPO on τ-bench and over an SFT checkpoint on 2WikiMultihopQA. Additional studies examine grouping choices, credit components, and the effect of missing tool feedback.

Critical strengths
  1. The reported τ-bench improvement is substantial. Average success increases from 27.57% with GRPO to 40.26% with Dr.Q, with gains in both retail and airline tasks. This is the strongest evidence that the proposed training approach can improve task completion in interactive tool-use settings.

  2. The discussion of missing tool feedback identifies a useful practical issue. Assigning zero reward to calls without recorded responses can penalize them relative to calls with positive feedback, even when the missing response does not indicate an actual failure. The replay analysis connects this concern to observed changes in tool use. This is relevant to researchers building training environments with incomplete feedback.

Critical weaknesses
  1. The contribution beyond existing credit assignment and process supervision methods is not sufficiently established. The method combines familiar elements: trajectory-relative advantages, discounted process rewards, and repeated-state comparisons related to GiGPO. Such a combination can be useful, but the experiments do not clearly separate the benefit of the credit assignment design from that of additional reward information. In particular, the τ-bench baseline is described as outcome-supervised GRPO, whereas Dr.Q includes rubric and recovery feedback. The reported gain therefore supports the full training approach, but provides less evidence for the specific credit assignment design. Further evidence separating these two effects would make the contribution more convincing.

  2. The evidence for the local credit components is incomplete. The paper does not report how often Anchor groups contain multiple distinct rollouts or how often their terminal outcomes differ. These statistics matter because singleton groups and groups with identical outcomes provide no Anchor signal. The component ablations mainly report training metrics, and the grouping setting is unclear for several configurations in Table 6. These gaps make it harder to judge the components’ contribution to held-out performance.

Questions for authors
  1. In the τ-bench experiment, do GRPO and Dr.Q use the same SFT initialization, training data, and rollout budget? Which process rewards are enabled for each method? If the reward information differs, could you provide one comparison in an existing setting where GRPO receives the same process rewards, for example by aggregating them into a trajectory score?

  2. Which macro grouping setting is used for each ablated configuration in Table 6, and are the other settings unchanged?

  3. Could you provide some statistics showing how often Anchor groups support a nonzero cross-rollout comparison during training, such as the distribution of distinct-rollout group sizes, the fraction of segments with a cross-rollout comparator, and the fraction of groups containing different terminal outcomes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.