acceptodds
Under review as a conference paper at ICLR 2027

TACT: Alleviating Credit Misassignment in Group-Based Policy Optimization for Search Agents with Tool-Aware Credit Transfer

Abstract

Mainstream group-based policy optimization methods broadcast trajectory-level advantages to all tokens, potentially penalizing useful steps in failed trajectories as heavily as unhelpful ones. We refer to this over-penalization of useful tool calls as credit misassignment. In search scenarios, many tool calls in failed trajectories have parameters similar to those of calls in successful trajectories for the same question, providing a signal for identifying and mitigating credit misassignment. Building on this insight, we propose Tool-Aware Credit Transfer (TACT), which leverages the parameter–intent correspondence of information-acquisition tools: similar call parameters provide probabilistic evidence of similar information-seeking intent, guiding conservative credit transfer. TACT constructs counterfactual references within the current training batch and attenuates potentially misassigned negative credit via confidence-gated conservative advantage correction. It requires no additional annotation, models, or sampling during training, and introduces negligible computational overhead. Across multiple search benchmarks, TACT delivers consistent, plug-and-play improvements over strong baselines for three group-based policy optimization algorithms (GRPO, GSPO, and SAPO). Our code and models will be publicly released upon acceptance.

Primary area:
reinforcement learning
Keywords:
Agentic RL, Search Agent, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QkvVOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TACT (Tool-Aware Credit Transfer), an advantage-correction mechanism that alleviates "credit misassignment" in group-based policy optimization (e.g., GRPO) for multimodal search agents. Existing methods broadcast a single trajectory-level advantage to every token, assuming all steps contribute equally. This breaks down in multi-step tool use: failed trajectories often contain tool calls that actually retrieved the needed information, yet they are penalized together with useless steps. Their key insight is that, for information-acquisition tools, the call parameters encode the retrieval intent, so successful matched calls serve as a counterfactual reference to compensate wrongly-penalized steps. Experiments on Qwen3-VL across 7 multimodal benchmarks show consistent gains.

Critical strengths
  1. Well-motivated problem. The misassignment phenomenon is real and quantified, making the diagnosis convincing.
  2. Plug-and-play and zero extra cost. TACT requires no annotation, no extra model.
  3. Broad evaluation: 7 multimodal benchmarks plus text-only validation, with fairly complete ablations.
Critical weaknesses
  1. Sensitivity of key hyperparameters (θ, α_min) and their cross-task transferability is not systematically analyzed.
  2. How matching is decided (rule/embedding/LLM) strongly affects results; matching errors propagate directly. Authors should discuss about it.
Questions for authors

How does the method degrade when successful trajectories are scarce and the reference bank is sparse?

Reviewer_f2CHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the issue of credit assignment errors in multi-step tool invocation reinforcement learning. The authors believe that in the failed trajectories, some tool invocations are actually beneficial and should not all be assigned negative advantage values. This method constructs a reference by matching similar parameter invocations in the successful trajectories, and performs conservative advantage value compensation for the corresponding steps in the failed trajectories.

Critical strengths
  1. The credit allocation in multi-step tool invocation is indeed a current pain point in the field of reinforcement learning. The author's observation of potentially beneficial steps in failed trajectories is worthy of recognition.

  2. This method does not require the introduction of additional reward models or complex Monte Carlo sampling. It only uses the data within the current batch for comparison, which indeed reduces some computational costs.

  3. The author demonstrated the hierarchical phenomenon of different tool types in terms of confidence score and credit transfer magnitude. This provides certain data support for understanding the behavior of the method in different scenarios.

Critical weaknesses
  1. This method heavily relies on parameter similarity to infer the consistency of information acquisition intentions. However, in actual complex search scenarios, the same query parameters may have completely different semantic values due to variations in the context state. The author performs credit transfer based solely on superficial parameter matching, which lacks solid theoretical support and is prone to introducing noise and even leading the model to learn incorrect strategies.

  2. The author mainly compares the method with the basic group relative strategy optimization and its simple variants, but deliberately avoids the more advanced and specifically designed latest methods in the current field that are dedicated to fine-grained credit allocation. This unfair comparison makes the claimed performance improvement seem unconvincing and unable to prove the true validity of the method.

  3. Although this method does not require additional model training, clustering, similarity calculation, and confidence gating filtering of all tool invocations within each training step inevitably bring significant time and memory overheads. As the batch size and complexity of tool invocations increase, this hidden computational burden will be dramatically amplified. The author downplays this and shows a serious lack of engineering rigor.

Questions for authors

see weakness.

Reviewer_vTckOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TACT addresses the credit misassignment problem in Agentic RL, where group-based policy optimization methods such as GRPO assign trajectory-level advantages to all tokens, potentially penalizing useful tool calls in failed trajectories. The core insight is that tool calls with similar parameters across successful and failed trajectories for the same question often share similar information-seeking intentions. Based on this observation, TACT constructs reference groups from successful trajectories and uses parameter similarity and successful-trajectory coverage to estimate confidence, conservatively compensating negative advantages for matched tool calls (capped at zero). The method requires no additional models, annotations, or sampling and can be seamlessly integrated into GRPO, GSPO, and SAPO. Experiments demonstrate consistent improvements across multimodal and text-based search benchmarks while mitigating entropy collapse and declining tool usage during training. The key contribution is a lightweight, cross-trajectory, tool-aware credit transfer mechanism that enables more fine-grained credit assignment at negligible computational overhead.

Critical strengths
  1. Well-Motivated Credit Misassignment Problem. The paper provides concrete evidence that over half of failed trajectories contain parameter-matched tool calls found in successful trajectories (Figure 2). This observation establishes a clear motivation for correcting trajectory-level credit assignment in GRPO.

  2. Effective and Low-Cost Credit Transfer Mechanism. TACT combines parameter similarity, successful-trajectory coverage, and conservative advantage clipping to selectively compensate negative advantages. The ablation results (Table 3) validate the importance of these components, while the method introduces only 0.06% additional training overhead.

  3. Comprehensive Experimental Evidence. TACT consistently improves GRPO, GSPO, and SAPO across seven multimodal search benchmarks (Table 2), with further improvements on text-only search tasks (Table 4). The transfer precision analysis (Table 5) also shows that 79.4% of transfer-triggered tool calls contain question-relevant information, providing direct support for the core hypothesis.

Critical weaknesses

See questions below.

Questions for authors
  1. Motivation (Figure 2). The paper identifies credit misassignment based on parameter-matched calls across successful and failed trajectories. However, does parameter similarity necessarily indicate incorrect penalization, given that identical tool calls may contribute differently depending on the reasoning context?

  2. Method (Eq. 2–5). Eq. (2) defines credit discrepancy using both successful and failed advantages, while Eq. (4) only considers successful advantages. What justifies this approximation?

  3. Experiments (Table 3). Could the authors compare TACT against random or uniform negative-advantage attenuation under a matched compensation budget to verify that the improvements come from accurate credit matching rather than simply reducing negative gradients?

2 comments

Thank you for recognizing the motivation, mechanism, and experiments of our work, and for suggesting equal-budget controls. The 79.4% versus 27.5% result in the independent tool-response evaluation supports the informational value of the selected calls. Eq. (2) analyzes the credit discrepancy, while Eq. (4) provides heuristic compensation, with the original advantage of the failing trajectory still retained. In the newly added equal-budget controls, TACT achieves an average accuracy of 59.71%, a relative improvement of 5.27% / 4.74% over Uniform / Random, with higher accuracy on all seven benchmarks. These results support the additional value of selective compensation; simply attenuating negative advantages does not reproduce all the gains. We explain each point below.

Q1: Does parameter similarity imply an incorrect penalty?

Brief summary: TACT matches information-seeking intent, rather than whether subsequent reasoning succeeds; higher similarity and greater support across successful trajectories provide a stronger structural basis for attenuating the negative advantages of actions in failing trajectories.

Parameter similarity does not necessarily imply an inappropriate penalty, but failed reasoning does not negate the value of necessary information. For the same question and tool type, parameters specify the information-acquisition target and can provide an intent-based reference for cross-trajectory comparison, rather than assuming identical contributions across contexts.

The independent tool-response evaluation shows that TACT can preferentially select useful calls. The proportions of triggered and non-triggered steps containing question-necessary information are 79.4% versus 27.5%; the proportion is 83.0% on the 265-step author-annotated subset, and the F1 score between the LLM and author annotations is 0.87. Figure 2 illustrates the matching phenomenon, while the tool-response evaluation examines informational value; neither is causal proof for each individual match.

To address retrieval noise or redundant evidence, TACT constrains compensation through weighting, gating, and a clamp at zero, only attenuating negative advantages without assigning positive advantages. In the evaluated GRPO, GSPO, and SAPO settings, average accuracy improves by approximately 4%–6% relative to the respective baselines.

Reviewer_vTckOpenReview ↗

Reply to the author

Thank you for your reply. I will keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.