acceptodds
Under review as a conference paper at ICLR 2027

SLCA-GRPO: Resolving Cross-Segment Credit Misattribution in Tool-Calling RL

Abstract

Tool-calling agents produce heterogeneous outputs, interleaving structured tool invocations with user-facing natural language summaries. This output heterogeneity presents a structural failure mode in standard on-policy Reinforcement Learning (RL): algorithms like GRPO indiscriminately broadcast a homogeneous trajectory-level scalar advantage to all tokens. Consequently, gradient noise from summary generation leaks into tool-decision tokens, causing cross-segment credit misattribution and brittle optimization. In this work, we propose SLCA-GRPO, a framework incorporating Segment-Locked Credit Assignment (SLCA). To enable scalable exploration without costly real APIs and stable training, we first construct the Schema-Guided LLM Simulator (SGLS) as foundational training infrastructure. Building on this, SLCA decouples advantage estimation at the structural segment level within a single group of rollouts, without requiring additional rollouts from intermediate states. Supported by Hierarchical Rewards (HierR), SLCA routes execution advantages to tool tokens and preference advantages to summary tokens, eliminating advantage contamination (the dominant cross-segment credit misattribution channel) within each policy update. On a 7B backbone, SLCA-GRPO accelerates convergence and outperforms standard GRPO, ToolPO, and RLTR by +2.53 pp on in-domain evaluation, +1.36 pp on the Berkeley Function-Calling Leaderboard (BFCL), and +9.15 pp on -Bench under the same training budgets, achieving higher accuracy with reduced tool redundancy and costs.

Primary area:
reinforcement learning
Keywords:
tool-calling agents, reinforcement learning, credit assignment, large language models, GRPO, tool use, on-policy learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_C2RnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies cross-segment credit misattribution as a structural flaw of GRPO for tool-calling agents: a shared trajectory advantage lets summary reward noise corrupt tool gradients, misassigning credit between tool-use and answer-generation segments. To address this within one unified model without extra rollouts, it proposes SLCA-GRPO. Using gradient masking, Hierarchical Rewards (HierR) and Segment-Locked Credit Assignment (SLCA), the method decouples, separately normalizes and exclusively routes tool and summary advantages to their corresponding tokens, isolating tool gradients from summary rewards. A schema-guided simulator SGLS enables low-cost large-scale RL without live APIs. With per-update theoretical isolation guarantees, SLCA-GRPO outperforms GRPO, ToolPO and RLTR across three backbones and benchmarks including BFCL and τ2-Bench and diagnostic and ablation experiments validate the identified failure and the necessity of SLCA, SGLS and HierR.

Critical strengths

This work proposes a novel training method targeting the credit assignment problem and validates its effectiveness.

Critical weaknesses
  1. Innovation is limit. Existing process-reward methods already support assigning position-wise differentiated rewards, which represents a conventional paradigm.
  2. The design of process rewards itself leaves substantial room for further investigation. The paper assigns low rewards to erroneous tool calls, yet this may be overly simplistic. For instance, an incorrect tool call within an otherwise complete reasoning chain, where the model detects the mistake and subsequently corrects it, may still carry learning value and should not necessarily receive a low score. Moreover, some redundant tool calls can serve as careful environmental verification to prevent downstream mistakes. Therefore, the criteria for judging correct/incorrect tool calls require more explicit elaboration.
Questions for authors
  1. Only single-round results on BFCL are reported.
  2. The empirical gains from the proposed credit assignment mechanism appear limited.
  3. What are the concrete rules used to judge whether a tool call is correct or incorrect?
Reviewer_yyqaOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

A tool-calling rollout is split by the loss mask into a tool segment (everything before the last model-generated run) and a summary segment (the final answer). Standard GRPO normalizes R_tool + R_sum jointly and gives every token the same advantage, so summary quality can reward or punish tool decisions. SLCA normalizes the two rewards separately and gives tool tokens only the tool advantage and summary tokens only the summary advantage. R_tool is a weighted match against gold tool calls (format, name, keys, values, first-step parallel count). R_sum is a GPT-OSS-120B judge score. Training uses an LLM-based tool simulator (SGLS). On Qwen2.5-3B/7B and Qwen3-8B trained on Toucan, the 7B model improves over GRPO by 2.53 on Toucan [email protected], 1.36 on BFCL single-turn, and 9.15 on tau2-Bench.

Critical strengths

The experiments are thorough for a paper of this type: three seeds, three scales, and a lot of controls in the appendix (a 2x2 support control, a reward-ratio sweep, a mocker swap, an execution-based reward). The tau2-Bench improvement over SFT is substantial on 7B and 8B.

Critical weaknesses
  1. Most of the gain comes from normalizing the two rewards separately, not from the routing. Table 21 has a "both open" condition: each reward is normalized separately, then summed and given to all tokens, with no routing. On 7B, going from unified GRPO to "both open" gives +5.55 on tau2, +0.89 on BFCL and +1.70 on Toucan. Adding the routing on top gives only +3.60 / +0.47 / +0.83, which is within the tau2 std (2-3 points) and under one point elsewhere. Normalizing multiple rewards separately is an existing technique, and that line of work is not discussed. The main "w/o SLCA" ablation changes both things at once, so the paper attributes the whole gap to routing. Also, the "T->S open" condition is as good as SLCA, so only one direction (summary reward reaching tool tokens) matters. That is a narrower claim than the one the paper makes. Related prior work such as MT-GRPO, which already assigns separately normalized turn-level and outcome advantages to different parts of the trajectory, should be a baseline.

  2. The method depends on gold tool trajectories, and the main metric is circular. Routing needs a tool reward that is independent of the final answer. In every main experiment that reward comes from matching gold calls in the Toucan teacher trajectories. It is a single trajectory-level score that ignores execution results (despite being called a "dense execution" / process reward). Gold demonstrations are exactly what is missing in the settings where RL is supposed to go beyond SFT. The only gold-free variant (D.12) uses an LLM "Solved/Unsolved" judge whose inputs are not described. If that judge reads the final answer, the tool reward is coupled to the summary again. In-domain, Toucan Success is defined as I[S_process >= 0.9], i.e. a threshold on the training reward, measured in the same simulator. The underlying differences are tiny (7B Process 0.8766 vs 0.8667, ArgMatch 0.8353 vs 0.8350). Because the 0.30-weight parallel term is binary, the threshold mostly tests whether the first step issues the same number of calls as the teacher.

Questions for authors
  1. Please report "both open" (separate normalization, no routing) as the main baseline, with significance over seeds.
  2. Why is SLCA below SFT on BFCL multi-turn for 3B and 8B?
Reviewer_cEjZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies "Cross-Segment Credit Misattribution" as a structural failure mode when applying trajectory-level RL (e.g., GRPO) to tool-calling agents: since rollouts decompose into a tool segment y_tool and a user-facing summary segment y_sum, broadcasting a single trajectory-level advantage to all tokens allows summary-reward noise to contaminate tool-token gradients (a failed tool call can be reinforced if followed by a good summary, and vice versa). The authors propose SLCA-GRPO with three components: (1) SLCA — segment-locked credit assignment that independently z-score-normalizes R_tool and R_sum within a rollout group and routes advantages exclusively to their corresponding token segments; (2) HierR — hierarchical rewards separating a dense tool-process score (format/name/key/value/parallelism) from an LLM-judged summary score; (3) SGLS — a schema-guided simulator (deterministic schema validation + frozen Qwen3-235B mocking) enabling scalable rollouts. On 7B, SLCA-GRPO reports +2.53pp in-domain (Toucan), +1.36pp (BFCL V3 single-turn), +9.15pp (τ2-Bench Pass1) over matched GRPO, with ablations, a reward-protocol sensitivity study, a support-control factorial experiment, and a local theoretical analysis (advantage isolation, nuisance-variance removal, directional fidelity in sign-conflict regimes).

Critical strengths

S1. A crisp, well-motivated problem. The failure mode is articulated precisely and verified constructively: the diagnostic examples (Fig. 1-style cases in §1, Fig. 9) show how a correct summary masks redundant/serial tool calls under a unified advantage, and the S_par=0.00 / R_sum=1.00 decomposition in Fig. 9 makes the misattribution directly visible. The framing of SLCA as a structural (execution vs. articulation) axis orthogonal to the temporal axis of VinePPO/GiGPO/SPO is a genuine conceptual clarification, honestly noting composability rather than claiming replacement.

S2. Method is simple, low-cost, and theoretically justified at the right level. SLCA changes only the scalar advantage attached to each token (Eq. 2–3) — no extra rollouts, no model split, no learned value function. The theory is correspondingly modest and honest: Prop. A.1 is a per-update functional-independence statement; Thm. A.3 is a conditional variance claim under an explicit nuisance-noise assumption; the paper repeatedly labels SLCA a bias–variance trade-off rather than an unconditional improvement. This restraint is rare and appreciated.

S3. Unusually careful experimental hygiene for this area. The matched GRPO/SLCA comparison fixes SFT init, data, simulator, rewards, G=16, decoding, and evaluation; results are three-run means ± std across three backbones (3B/7B/8B). The support-control factorial (Table 21–22) cleanly decomposes the routing effect (closing S→T path: +5.51pp τ2-Bench), separating it from the normalization confound in the naive w/o-SLCA ablation. The unified-reward ratio sweep (Table 20) preempts the obvious "just re-weight the unified reward" rebuttal. The ToolPO protocol-sensitivity study (Table 9), the response-mocker swap (Table 8), and the execution-based R_succ check (Table 25) further probe robustness to reward/mocking choices.

S4. Positive transfer to judge-free, dynamic benchmarks. The τ2-Bench gains (+9.15pp / +10.03pp on 7B/8B) are on an official, judge-free protocol with a DeepSeek-V3.2 user simulator, so the headline improvement is not purely an artifact of the paper's own reward/eval stack.

Critical weaknesses

W1. The core evaluation metric is circular: the training reward is the test metric. The primary in-domain metric (Toucan "Success" = I[S_process ≥ 0.9]) is the same S_process used as the dense training reward in HierR (Eq. 22), and NameF1/ArgMatch are its sub-components. SLCA optimizes R_tool directly, so the +2.53pp in-domain gain partially measures reward overfitting rather than tool-use capability. The paper anticipates this (Limitations §6; BFCL/τ2 are judge-free), and Table 25 (execution-based R_succ) helps, but the paper still leads with the circular number in the abstract and Table 1. At minimum, the abstract/conclusion should foreground the judge-free results.

W2. The τ2-Bench headline gap is fragile across scales and domains. +9.15pp (7B) and +10.03pp (8B) shrink to +1.09pp on 3B; per-domain (Table 18) the ordering flips (e.g., 3B: SLCA trails GRPO on Retail; 7B: trails on Airline; 8B: ToolPO beats SLCA on Retail and is within 0.37pp on Overall). τ2 uses Pass1 with a single trial per task on a finite task set, so per-domain numbers are discrete proportions with high variance; no significance testing is reported anywhere in the paper. With three runs and std ≈ 1–3pp, several claimed gaps (e.g., 3B BFCL +0.40pp with σ ≈ 0.3–0.5pp) are within noise.

W3. Baseline ToolPO is arguably strawmanned. The 7B ToolPO row in Table 1 (Success 0.200) uses an LLM-judge outcome reward that causes a well-documented format collapse (format_passed=38%, App. C.4/D.3), while the reference-rule version scores 0.7735 (Table 9) — close to GRPO. Presenting the collapsed configuration in the main table, with the salvage only in the appendix, overstates the margin over prior work. The authors disclose this transparently, but the main-table choice still shapes the narrative.

W4. The theory does not cover the algorithm actually run. Thm. A.3 analyzes a local score-function estimator after "suppressing group mean and standard-deviation terms" (App. A.2), while the implementation z-score-normalizes per segment within the group — normalization makes advantages cross-sample dependent, and the removed-variance claim does not hold exactly for the deployed estimator. The authors say so, but then Prop. A.1's "advantage isolation" (Eq. 4) is also diluted: in group normalization, A_hat_tool of rollout i depends on R_tool of all rollouts, and A_hat_sum depends on summary rewards of all rollouts — including rollouts whose summaries were contaminated. The clean statement holds only for the un-normalized estimator.

W5. The simulator (SGLS) creates a train/eval-distribution gap that is only partially quantified. Training rollouts interact with mocked tool responses; τ2-Bench and BFCL use real/official environments. The paper offers alignment examples (Fig. 7) and the w/o-SGLS ablation, but no quantitative measure of simulator fidelity, and no experiment isolating simulator-realism effects on the SLCA vs. GRPO gap specifically. The Qwen3-235B-A22B mocker generating observations for a Qwen2.5/3 policy raises a "distribution curated by a same-family frontier model" concern that is mitigated (cross-family, GPT-OSS-120B swap) but not eliminated.

W6. Scope limitations that bound impact. (i) Only ≤8B models and ~100 RL steps (~1 epoch, 31.8k prompts) — the gains appear within a very short budget; it is unclear whether the cross-segment contamination matters at longer horizons or larger scales. (ii) SLCA assumes a clean tool/summary boundary; multi-turn dialogues with interleaved summaries or free-form code generation break the assumption (acknowledged in §6). (iii) The in-domain result depends on gold-trajectory matching, which under-rewards alternative valid plans (acknowledged). (iv) The claimed orthogonality to temporal credit assignment is asserted but never tested — even a small SLCA+GiGPO/VinePPO combination experiment would substantiate the composability claim.

Questions for authors

Q1. In Table 18, why does the τ2-Bench gain concentrate at 7B/8B (+9.15/+10.03pp) but nearly vanish at 3B (+1.09pp)? Is this a capability threshold (weaker models produce uniformly bad tool segments, so z-scored A_tool carries little signal) or a variance artifact of the single-trial Pass1 protocol? Q2. Under group normalization, A_hat_tool(i) is a function of {R_tool(j)} for all j in the group. Given Prop. A.1's per-update isolation claim, can you state precisely which dependency the deployed (normalized) estimator does and does not break, ideally as a revised proposition? Q3. The in-domain gains materialize within ~100 policy-update steps on 31.8k prompts. How do SLCA vs. GRPO compare at 3–5× the RL budget — does the gap persist, widen, or close (i.e., is the contamination a transient or persistent optimization pathology)? Q4. For trajectories where the summary is factually dependent on tool quality (the judge's "Information Completeness" dimension), SLCA still locks A_sum to be independent of A_tool routing. Have you measured whether summary quality degrades relative to GRPO on tasks where summarization requires deep reasoning over tool outputs (Table 15 shows GRPO wins Summary on 3B/8B — is that the mechanism)? Q5. The w/o-SGLS ablation removes schema conditioning and deterministic validation but keeps the same mocker. Have you tried the reverse (keep validation, replace the LLM mocker with, e.g., canned/gold responses) to separate "mock realism" from "mock structure"? Q6. Given the claimed orthogonality to temporal credit assignment, even a small-scale SLCA + GiGPO (or VinePPO) combination on 7B would directly test composability. Was this attempted, and if not, why?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.