acceptodds
Under review as a conference paper at ICLR 2027

Efficient Agentic Reinforcement Learning with Tool-Aware Credit Assignment

Abstract

Agentic reinforcement learning (RL) equips large language models (LLMs) with tool-use capabilities that substantially improve reasoning on complex tasks. However, incorporating external tool feedback perturbs the model's predicted token distribution, introducing distribution shifts that destabilize policy optimization. To address this issue, we propose TACT, an efficient framework that stabilizes policy learning through tool-aware credit assignment. Specifically, to quantify a tool's impact on the reasoning process, TACT performs counterfactual reasoning by masking tool observation tokens. By computing the information gain between the original trajectory and its counterfactual counterpart, we establish a rigorous token-level metric for credit allocation. Guided by this metric, TACT hierarchically refines the optimization process. At the trajectory level, executions with failed tool outcomes are filtered out to prevent erroneous feedback from exacerbating distribution shifts. At the token level for successful trajectories, the information gain serves as the criterion for shaping token-level advantages, allocating higher credit strictly to tokens where tool interaction meaningfully advances the final outcome. By aligning policy updates with validated tool contributions, TACT mitigates the error interference from failed trajectories while reinforcing the effect of tokens with higher information gain. Extensive experiments on 9 benchmarks spanning mathematical reasoning, code generation, and function calling, across 3 model scales, demonstrate the superiority of TACT over existing methods.

Primary area:
reinforcement learning
Keywords:
agentic reinforcement learning, large language model, reasoning, credit assignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FG5YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TACT, a modification of GRPO that combines trajectory filtering with token-level advantage reweighting. A second forward pass masks tool observations, and the resulting token log-likelihood difference determines how much to amplify successful post-tool reasoning. Experiments cover three Qwen checkpoints, seven mathematics benchmarks, and additional code-generation and function-calling evaluations. The authors report accuracy improvements, fewer tool calls, and modest additional training cost.

Critical strengths

The improvements are measured against SimpleTIR, not only against the unstable standard-TIR baseline. The reported mathematics average increases from 41.00 to 43.69 for the 4B model and from 40.21 to 42.51 for the 7B model. The reductions in tool calls are also useful, although they should not be equated with reductions in total generation cost.

Critical weaknesses
  • The evidence does not yet establish that the proposed signal improves tool-specific credit assignment rather than simply reweighting successful continuations. Equation (5) measures an observed token’s likelihood sensitivity to masked feedback; this need not indicate that the feedback helped solve the task. Appendix D.7 varies the gating scope, but does not isolate the information in the token weights from their average amplification. A constant or shuffled-weight control under identical filtering would address this central attribution issue.

  • The stated objective and pseudocode disagree about unsuccessful trajectories. Equation (9) and Appendix B retain the original advantages for valid but unsuccessful rollouts. Appendix H instead defines valid_mask = (seq_scores > 0) and multiplies the final loss mask by it, removing those rollouts from optimization altogether. This changes the learning objective, not merely the IG gate, and leaves unclear which method produced the reported results.

Questions for authors
  • Which loss mask was used for Tables 1–2? Please reconcile Equation (9), Appendix B, Algorithm 1, and Appendix H, and specify whether SimpleTIR used the same trajectory filtering.

  • With filtering and the success/post-tool gate fixed, does IG weighting outperform constant or shuffled multipliers matched for average amplification? This would distinguish the proposed attribution signal from generic positive-example reweighting.

8 comments
AuthorsOpenReview ↗

Response 1 for Reviewer FG5Y

We thank the reviewer for the insightful and encouraging comments, which have been invaluable in improving our manuscript. We have addressed each weakness and question below, and revised the main paper and appendix accordingly, with all changes marked in red.

W1&Q2: Does IG weighting outperform constant or shuffled multipliers matched for average amplification?

R1: We have added ablations that isolate the contribution of the IG signal. All variants share TACT's trajectory filtering, gating and hyperparameters, and differ only in the multiplier applied to gated tokens:

  • Constant: every gated token receives the same multiplier, equal to the batch mean of MtM_t, which matches TACT's average amplification at every step.
  • Shuffled: the MtM_t values are randomly permuted across gated tokens within each batch. This preserves both the mean and the distribution of amplification, but removes the alignment between IG and individual tokens.

IG alignment matters beyond amplification magnitude. The mean-matched constant multiplier recovers only a small part of the gain (40.39 Avg@16). The shuffled multiplier, which keeps the same mean and distribution of amplification, reaches 41.73. TACT still outperforms it by +0.78 Avg@16 and +4.10 Pass@16. Both controls receive the same average amplification as TACT, so the remaining gap comes from which tokens are amplified, not from a generic upweighting of positive examples.

The benefit is largest on the hardest benchmarks, such as Hmmt25 (+2.29 Avg@16, +3.33 Pass@16) and AIME25 (+1.04 / +3.33), where correct solutions depend more heavily on using intermediate tool results.

In summary, these results confirm the effectiveness of IG: the gains of TACT come from IG-guided, token-level credit assignment rather than from uniform advantage amplification.

Table R1(a). Pass@16 of IG ablations on Qwen2.5-7B. Best results are in bold.

MethodAIME24AIME25MATH500OlympiadAMC23Hmmt25MinervaAvg.
Constant70.0046.6794.8067.6092.5040.0038.2464.26
Shuffled56.6750.0097.0069.8092.5036.6739.3463.14
TACT70.0053.3395.4069.2097.5043.3341.9167.24
Δ+0.00+3.33−1.60−0.60+5.00+3.33+2.57+2.98

Table R1(b). Avg@16 of IG ablations on Qwen2.5-7B. Best results are in bold.

MethodAIME24AIME25MATH500OlympiadAMC23Hmmt25MinervaAvg.
Constant32.9222.2978.2244.8066.2516.0422.2440.39
Shuffled33.1223.9679.4646.4269.5316.4623.1641.73
TACT33.7525.0079.3346.6570.4718.7523.6442.51
Δ+0.63+1.04−0.13+0.23+0.94+2.29+0.48+0.78

Revised sections: Appendix D.7 (Ablation Analysis).

AuthorsOpenReview ↗

Response 2 for Reviewer FG5Y

W2&Q1: Method inconsistency

R2: We thank the reviewer and apologize for the confusion. All results in Tables 1, 2 come from the same training procedure, which matches Sec 4.3, Appendix B.1 and Algorithm 1:

  • Loss mask. The token-level loss mask excludes prompt tokens and tool-observation tokens. Valid but incorrect trajectories in retained groups keep their original group-relative advantage and still contribute negative-advantage updates.

  • Advantage shaping. Within valid trajectories, MtM_t is applied only when the triple-gating condition $G_t \gets \mathbb{1}[\text{valid}(o)] \cdot \mathbb{1}[\text{success}(o)] \cdot \mathbb{1}[t \in \mathcal{I}_{\text{post}}] $ holds. Otherwise, the original advantage is retained (Eq. 9). Valid but incorrect rollouts therefore fall into the otherwise branch: they keep their original negative group-relative advantage and still contribute to the policy loss.

Reconciliation with Appendix H. The previous pseudo-code contained two errors: valid_mask = (seq_scores > 0) is in fact the success indicator, which is used only for the IG gate. final_loss_mask = loss_mask * valid_mask was erroneous and has been removed. We have rewritten Appendix H to separate validity from success explicitly. SimpleTIR uses its original void-turn filtering. For format-related filtering, TACT follows the same criterion as SimpleTIR.

Revised sections: Sec 4.3, Appendix B.1, Appendix C.1, Algorithm 1, Appendix H (Policy Optimization).

Thanks for the revisions and new experiments. The updated Appendix H resolves my concern about valid-but-wrong trajectories being excluded from the loss. The constant and shuffled ablations also directly answer my request and offer solid evidence for token-wise weighting. Two issues still stand.

  1. The implemented IG does not cleanly isolate tool feedback. As noted in Appendix B.1, factual log-probs are cached from the old policy, while counterfactuals are recomputed with the current actor before each mini-batch update. After the first update, their difference conflates policy drift and masked-observation effects — unlike the same-policy comparison in Eq. (5). The constant/shuffled baselines cannot tease these apart. Please quantify this mismatch, or run the comparison with both terms evaluated under a fixed policy. This is critical for interpreting the method as tool-aware credit assignment.
  2. The IG gains lack clear reliability. Tables 10–11 show +0.78 Avg@16 and +4.10 Pass@16 over shuffled weighting, with no mention of independent training repeats or uncertainty for these gaps. The three BFCL evaluation runs only cover evaluation noise, not training variance. Please share the seeds used for these ablations and paired per-seed results if you have them. The numbers are promising, but they do not yet prove the edge over random weighting is reproducible.

The revision is much better, and I am no longer worried about the loss masking issue. My main hesitation is whether the implemented signal and reported gains support the central credit-assignment claim. I will keep my score at 2.

Reviewer_NAqgOpenReview ↗
Rating
2(+1)
Confidence
4(+1)
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

TACT address distribution shift during policy optimization caused by tool integration. It modifies GRPO with a code interpreter under TIR settings. TACT purpose that an additional forward pass replaces tool observations with padding tokens, and the difference in the sampled original token log-probabilities supplies a bounded advantage multiplier. Structural filtering and success gating determine where it applies. Experiments compare TACT and SimpleTIR on seven mathematics benchmarks and three Qwen backbones, with transfer evaluations on LiveCodeBench and BFCL.

Critical strengths
  1. The main accuracy gains are reasonably consistent across backbones. Table 1 improves the seven-benchmark average from 41.00 to 43.69 for Qwen3-4B and from 40.21 to 42.51 for Qwen2.5-7B. Figure 6 improves LiveCodeBench Avg@16 from 20.36 to 25.13. These gains over a competitive baseline, together with transfer to coding, support practical value.

  2. The method focus on distribution shift problem on Tool-use and the method does not require training another model, except policy itself. App.D.6 measures an extra 16.52 seconds on a 384.36-second baseline step, or 4.30% overhead. This modest measured training cost makes the approach worth investigating, although fewer tool calls alone do not establish lower total inference latency.

Critical weaknesses

W1 : For presesntaion, Information Gain as criticial terminology is shorted as 'IG' without declear in main context. I found it myself in App.G line 1316-1317. In addition, the executed optimization is unclear. Sec.4.3, B.1, and Algorithm 1 distinguish structural validity from answer success and retain the baseline advantage for valid but incorrect rollouts. App.H instead sets valid_mask = (seq_scores > 0) and final_loss_mask = loss_mask * valid_mask. In a structurally valid group with binary rewards, the algorithm retains the incorrect rollout's negative-advantage update; H removes its entire policy loss. These are materially different training procedures. H also computes old_log_probs - cf_log_probs, with the second term produced by self.actor; Algorithm 1 specifies the old policy for both terms. The subtraction direction agrees, but the parameter and cache timing needs explanation.

W2 : I would state that novelty is limited. The contribution beyond the closest estimator and the causal meaning of the signal are insufficiently established. InfoPO [1], already cited in §4.2, also scores the same realized continuation under factual and masked feedback; it discusses PAD/mask choices, and it also uses group-relative policy optimization. TACT's potentially useful differences are masking all trajectory observations together, token-level rather than turn-averaged shaping, and gated multiplicative rather than additive advantages. Those differences need a matched comparison; PAD and GRPO themselves do not establish novelty.

Reference: [1]. InfoPO, https://arxiv.org/html/2603.00656v2

Questions for authors

Q1: Which loss mask was used for Table 1? Does a valid zero-reward rollout in a mixed-reward group receive a negative-advantage update? Please give the actual reward/validity definitions and reconcile H with Algorithm 1. At what point are factual and counterfactual log-probabilities cached, and do they use the same policy parameters?

Q2: Under matched tasks, tools, filtering, and budgets, what improvement remains over an adapted InfoPO, alpha = 0, and a constant multiplier of comparable magnitude on successful post-tool tokens? Which proposed difference explains the gains?

Q3: What exact task IDs, effective denominator, sampling counts, and averaging procedure produced Olympiad Table 5? C.2 lists 674 tasks and D.1 describes any-correct among 16 samples. Under one evaluation of all 674 tasks, 75.60% and 69.20% cannot be obtained by rounding an integer success count: the nearest fractions are 510/674 = 75.6677% and 466/674 = 69.1395%. A filtered subset, repeated evaluations, or a pass@k estimator with more than 16 samples would change this check, so please state which was used.

Q4: Are Figures 7 and 15 different BFCL subsets/checkpoints? Besides Parallel 35→54 versus 35→51, Multiple is 41→42.5 versus 41→49. Please supply their task/version mapping and correct D.8's 19-point claim for Figure 15. Does Figure 8 use the same outputs as Tables 1/5, and are its different K values based on nested samples?

8 comments
AuthorsOpenReview ↗

Response 1 for Reviewer NAqg

We thank the reviewer for the insightful and encouraging comments, which have been invaluable in improving our manuscript. We have addressed each weakness and question below, and revised the main paper and appendix accordingly, with all changes marked in red.

W1&Q1: Information Gain & unclear optimization

R1: We address each point below.

1.We apologize for the omission. "Information Gain (IG)" is now defined at its first occurrence in the main text.

2.Executed optimization. All reported results come from a single training procedure, which matches Sec 4.3, Appendix B.1 and Algorithm 1. Appendix H contained two errors, which we have corrected: Trajectory-level filtering is performed during batch construction. Groups with identical rewards are discarded via group-level rejection sampling. Rollouts with failed tool executions are removed. The loss mask only excludes prompt and tool-observation tokens at the token level. Structurally valid but incorrect rollouts therefore keep their original group-relative advantage and still contribute negative-advantage updates, consistent with Eq. (9).

In Appendix H, valid_mask = (seq_scores > 0) is in fact the success mask and serves only the IG gate. The line final_loss_mask = loss_mask * valid_mask was erroneous and has been removed. We have rewritten Appendix H to separate validity and success explicitly.

3. Parameters and timing of the two log-probability terms.

old_log_probs are computed by the actor at the start of each GRPO iteration and cached. They stay fixed throughout the iteration and also serve as the denominator of the importance ratio. The counterfactual log-probabilities are computed by a no-grad forward pass with the current actor weights, immediately before each mini-batch's training forward pass. They are evaluated on the same response tokens at the same sampling temperature, and the difference is detached immediately.

Hence both terms come from the same weights before the first parameter update of each iteration. After that, the counterfactual term may reflect the slightly updated policy. We note that: (i) IG only rescales the advantage. It enters neither the importance ratio nor the gradient. (ii) Within one iteration, policy drift is limited by clipping, and the multiplier is further bounded by the ReLU, the clip at S_max and α. We have revised Algorithm 1 accordingly.

Revised sections: Sec 1 (IG definition), Appendix B.1(Detail of TACT), Algorithm 1, Appendix H.

AuthorsOpenReview ↗

Response 2 for Reviewer NAqg

W2&Q2: Comparison with InfoPO[1] and Novelty

R2: We thank the reviewer for this careful comparison. We clarify the relationship below, and we have added a matched comparison and a dedicated discussion to the revision.

1. Shared theoretical foundation.

Both methods measure how much an external observation shifts the policy's likelihood of a realized continuation, via a log-likelihood ratio between factual and masked contexts. This quantity is not specific to either work: it is a pointwise conditional mutual information, and counterfactual masking is a standard tool in information-theoretic and causal attribution [2–5]. We do not claim the estimator itself as a contribution. Our contribution is to turn this signal into a tool-aware, token-level credit assignment mechanism for tool-integrated RL. TACT credits the tokens that use tool outputs, rather than the action that elicited the feedback. It applies this credit as a bounded multiplicative rescaling of the outcome advantage, rather than as an additional reward.

2. Where TACT differs.

The difference lies not in how the information gain is computed, but in three design questions: to whom the signal is credited, how it enters the advantage, and on which trajectories it acts. These choices follow from a different problem setting.

  • InfoPO addresses underspecified user intent. For example, a user asks to book a flight without giving dates or budget, and the agent must ask questions to clarify. It credits the action that elicits informative feedback.

  • TACT addresses tool-integrated reasoning. For example, the model runs Python and continues reasoning from the output. Failed or misleading outputs shift the reasoning distribution and destabilize training. It credits the tokens that use valid tool outputs.

InfoPOTACT
SettingUser-centric multi-turn interaction (intent elicitation)Tool-integrated reasoning (code execution)
Core problemSparse terminal rewards; zero-variance groups give no gradientErroneous or noisy tool feedback induces distribution shift and destabilizes optimization
Masked contextOnly the most recent feedback o_t, one counterfactual per turnAll tool observations jointly, in a single counterfactual pass
GranularityTurn-level (averaged over the next action segment)Token-level (each post-tool token)
Credit assigned toThe action that elicited the feedbackThe tokens that consume the tool output
Role of IGAn additional intrinsic reward, with its own normalized advantageA rescaling factor on the existing outcome advantage; no new reward is introduced
FusionAdditive, variance-gated: A_ext + β·g(σ)·A_infoMultiplicative, positive-only: (1 + α·S_t)·A_t
Trajectory scopeAll trajectories, including zero-variance groupsValid and successful trajectories only (hierarchical filtering)
Extra cost~1.63× wall-clock (reported in [1])+4.3% per step (Appendix D.5)

3. Why these choices suit tool-integrated reasoning.

  • Crediting consumption rather than elicitation. In user-centric tasks, the decision that matters is what to ask. In tool-integrated reasoning, a tool call is only valuable if the subsequent reasoning actually uses its output; for example, the model copies the computed value "385" into its derivation (Fig. 2). Token-level credit on post-tool tokens targets exactly this.

  • Joint masking. Reasoning after the k-th tool call typically conditions on all earlier execution results, such as intermediate values reused across calls. Masking all observations measures each token's dependence on the full external context. This matches our goal: we want to separate what the model derives from tool feedback from what it generates on its own. Attributing credit to an individual call is a different goal.

  • Multiplicative, positive-only shaping. InfoPO's additive intrinsic reward is designed to create gradients where the outcome signal vanishes. In tool-integrated reasoning, the main failure mode is different: failed or malformed tool outputs inject noise. An intrinsic reward computed on such trajectories risks reinforcing it, and our scope ablation (Table 12, w/ Scope: failed) is consistent with this. TACT therefore never creates a learning signal independent of the outcome. It only redistributes credit within successful, valid trajectories. This also means it cannot reward unnecessary tool calls, consistent with the reduction in tool calls we observe.

AuthorsOpenReview ↗

Response 3 for Reviewer NAqg

4. Empirical evidence on token-level credit assignment.

InfoPO computes the information gain of each token in the next action and then averages it over the whole action segment. As a result, all tokens within the segment receive the same IG-based credit, regardless of how much each individual token depends on the feedback. To emulate this averaged treatment within our framework, we construct a Constant variant: every gated token receives the same multiplier, equal to the batch mean of MtM_t. This preserves the average strength of the IG signal but, like InfoPO's segment-level averaging, removes token-level differences in credit. We compare it with TACT on Qwen2.5-7B under identical filtering, gating and hyperparameters, so that the only difference is whether IG is applied uniformly or per token.

As shown below, TACT outperforms the averaged-IG variant on all seven benchmarks in Avg@16 (+2.12 on average), and matches or exceeds it on all seven in Pass@16 (+2.98 on average). The gains are largest on challenging benchmarks such as AMC23 (+4.22 Avg@16, +5.00 Pass@16) and AIME25 (+2.71 Avg@16, +6.66 Pass@16), where solutions rely most on intermediate tool results. This shows that the benefit of TACT does not come from the overall strength of the IG signal, but from assigning credit to the specific tokens that depend on tool outputs, which supports our token-level design over averaged credit assignment. Full results are reported in Appendix D.7.

Table R2(a). Avg@16 on Qwen2.5-7B. Best results are in bold.

MethodAIME24AIME25MATH500OlympiadAMC23Hmmt25MinervaAvg.
w/o IG32.5021.2577.2842.9868.7515.2121.8039.97
Constant32.9222.2978.2244.8066.2516.0422.2440.39
TACT (token-level IG)33.7525.0079.3346.6570.4718.7523.6442.51
Δ+0.83+2.71+1.11+1.85+4.22+2.71+1.40+2.12

Table R2(b). Pass@16 on Qwen2.5-7B. Best results are in bold.

MethodAIME24AIME25MATH500OlympiadAMC23Hmmt25MinervaAvg.
w/o IG56.6746.6795.0068.6092.5033.3339.7161.78
Constant70.0046.6794.8067.6092.5040.0038.2464.26
TACT (token-level IG)70.0053.3395.4069.2097.5043.3341.9167.24
Δ+0.00+6.66+0.60+1.60+5.00+3.33+3.67+2.98

Revised sections: Appendix D.7 (Ablation Analysis), Appendix D.10 (Comparison with InfoPO)

[1]Kong F, Zhang J, Deng M, et al. InfoPO: Information-Driven Policy Optimization for User-Centric Agents. ICML, 2026.

[2] Cover T M, Thomas J A, Kieffer J. Elements of information theory[M]. New York: Wiley, 1991.

[3] Church K, Hanks P. Word association norms, mutual information, and lexicography[J]. Computational linguistics, 1990, 16(1): 22-29.

[4] Li J, Monroe W, Jurafsky D. Understanding neural networks through representation erasure[J]. arXiv preprint arXiv:1612.08220, 2016.

[5] Cohen-Wang B, Shah H, Georgiev K, et al. Contextcite: Attributing model generation to context[J]. Advances in Neural Information Processing Systems, 2024, 37: 95764-95807.

Q3: Result in OlympiadBench

R3: We thank the reviewer for this careful check. The discrepancy comes from a fixed cap on evaluation size, not from a computational error. We have now added a detailed specification of the protocol to Appendix C.1, summarized as follows.

Evaluation protocol. For each benchmark, we evaluate at most 500 problems. Benchmarks with more than 500 problems are truncated to the first 500 in their fixed original order; smaller benchmarks are evaluated in full. OlympiadBench (674 problems) is the only benchmark affected by the cap. Its effective denominator is therefore 500. MATH500 (500), Minerva (272), AMC23 (40), AIME24/25 (30 each) and HMMT25 (30) are evaluated in full. The same 500-problem subset and protocol were used for all methods, so the comparisons remain matched. The raw counts for OlympiadBench Pass@16 (Table 5) are reported below.

OlympiadBench Pass@16 (Table 5), as raw counts:

ModelMethodSolvedUnsolvedTotalPass@16 (%)
Qwen3-4B-BaseSimpleTIR36413650072.80
Qwen3-4B-BaseTACT37812250075.60
Qwen2.5-7BSimpleTIR33316750066.60
Qwen2.5-7BTACT34615450069.20
Qwen3-8B-BaseSimpleTIR38311750076.60
Qwen3-8B-BaseTACT38211850076.40

Revised sections: Appendix C.1 (Evaluation protocol).

Reviewer_yC9UOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

TACT is an agentic RL framework for tool-integrated reasoning that addresses the uniform credit assignment problem in GRPO-based training. It constructs a counterfactual trajectory by masking tool-observation tokens with [PAD], computes token-level information gain (IG) as the log-probability difference between actual and counterfactual predictions, and uses a bounded IG signal to hierarchically reshape advantages: trajectory-level filtering removes failed rollouts, and token-level triple-gating amplifies advantages only for tokens that are causally post-tool, from successful trajectories, and have positive base advantage.

Critical strengths

Well-motivated and principled method. The counterfactual IG formulation gives a causal-inference-grounded proxy for tool utility without auxiliary reward models, directly targeting the uniform-credit bottleneck of GRPO in agentic settings.

Strong efficiency evidence. TACT achieves higher accuracy with substantially fewer tool calls and shorter generated code (Figure 5, 12, 13), and the "longer reasoning + less tool redundancy" pattern is a compelling narrative for learned tool discipline.

Critical weaknesses
  • Missing most-relevant baselines. Despite discussing GRPO-RoC and GiGPO as direct competitors in Related Work, no empirical comparison is provided (e.g., Retool, ToRL); only TIR and SimpleTIR are evaluated, weakening the "superior to existing agentic RL methods" claim.

  • Method inconsistency between main text and appendix. Eq. 8 defines the gate as I[valid]·I[success]·I[t∈I_post], but Algorithm 1 adds I[A_t>0]. This is a substantive discrepancy affecting whether negative-advantage tokens are amplified, and must be reconciled.

Incomplete ablation isolating IG's contribution. No ablation removes the token-level multiplier while keeping trajectory filtering, nor replaces IG with a random multiplier;

Unexplained regressions and cross-family gap. TACT underperforms SimpleTIR on 8B OlympiadBench (−0.78) and 4B AMC23 (−0.94) without analysis, and all experiments use Qwen models only, limiting the generality claim.

Questions for authors

see weakness

8 comments
AuthorsOpenReview ↗

Response 1 for Reviewer yC9U

We thank the reviewer for the insightful and useful feedback. All of them are important for further improving our manuscript. We have addressed each weakness and question below, and revised the main paper and appendix accordingly, with all changes marked in red.

W1: Missing most-relevant baselines.

R1: We have added two representative agentic RL methods, ReTool[1] and ToRL[2], as baselines. All methods use the same base model of Qwen2.5-7B, training data and number of training steps, and method-specific hyperparameters follow the original papers. As shown in Table R1a and R1b below, TACT achieves the best Avg@16 on all seven benchmarks. Its average (42.51) exceeds ToRL by +10.88 and ReTool by +11.68. And, TACT achieves the best average Pass@16 (67.24 vs. 61.21 for ToRL and 58.96 for ReTool), and the best result on six of the seven benchmarks. ReTool is slightly higher on OlympiadBench (70.80 vs. 69.20). ToRL and ReTool train on full trajectories without filtering failed tool executions, and they fall clearly below TACT.

Table R1(a). Avg@16 on Qwen2.5-7B. Best results are in bold. Δ denotes the difference between TACT and the better of ToRL and ReTool.

MethodAIME24AIME25MATH500OlympiadAMC23Hmmt25MinervaAvg.
ToRL20.2116.0467.4138.0550.9412.5016.2531.63
ReTool14.7911.4671.7637.5752.666.8820.7030.83
TACT33.7525.0079.3346.6570.4718.7523.6442.51
Δ+13.54+8.96+7.57+8.60+17.81+6.25+2.94+10.88

Table R1(b). Pass@16 on Qwen2.5-7B. Best results are in bold.

MethodAIME24AIME25MATH500OlympiadAMC23Hmmt25MinervaAvg.
ToRL53.3350.0093.8068.6087.5036.6738.6061.21
ReTool53.3340.0094.0070.8087.5026.6740.4458.96
TACT70.0053.3395.4069.2097.5043.3341.9167.24
Δ+16.67+3.33+1.40−1.60+10.00+6.66+1.47+6.03

Tool-Use Reasoning Analysis. We further analyze the average response length (Table R1c) and the average number of code lines per response (Table R1d) on each benchmark. TACT writes 50.72 code lines per response on average, 40.7% more than ToRL, and its response lengths are 31.5% and 18.4% longer, respectively. The increase is concentrated on the hardest benchmarks: on Hmmt25 and AIME24, TACT adds +26.66 and +24.00 code lines over ToRL (71.23 vs. 44.57 and 67.51 vs. 43.51), whereas on the easier MATH500 and Minerva, the increase is much smaller (+3.30 and +7.66 lines, i.e., 27.03 vs. 23.73 and 32.39 vs. 24.73). Its response length also grows mainly on AIME25 and HMMT25 (about +430 over ReTool). TACT therefore adapts tool use and reasoning length to problem difficulty, rather than becoming uniformly verbose. This behavior follows from tool-aware credit assignment: by amplifying the advantage of post-tool tokens that draw on tool observations in successful trajectories, TACT reinforces reasoning that actually uses tool results. In addition, the corresponding training curves are provided in Appendix D.1 of the revised paper.

Table R1(c). Average response length on Qwen2.5-7B. Δ denotes the difference between TACT and the higher of ToRL and ReTool.

MethodAIME24AIME25MATH500OlympiadAMC23Hmmt25MinervaAvg.
ToRL1441.921631.46921.941188.501190.381405.151064.651263.43
ReTool1704.601712.041020.791371.951385.151546.911081.131403.22
TACT2043.852147.461091.551562.791422.561977.951387.621661.97
Δ+339.25+435.42+70.76+190.84+37.41+431.04+306.49+258.75

Table R1(d). Average number of code lines per response on Qwen2.5-7B base model.

MethodAIME24AIME25MATH500OlympiadAMC23Hmmt25MinervaAvg.
ToRL43.5152.0023.7331.1332.5944.5724.7336.04
ReTool8.3412.136.498.679.0115.325.219.31
TACT67.5170.2427.0345.4941.1371.2332.3950.72
Δ+24.00+18.24+3.30+14.36+8.54+26.66+7.66+14.68

Related Revised Sections: Appendix D.1 (More Comparison)

AuthorsOpenReview ↗

Response 2 for Reviewer yC9U

W2: Algorithm inconsistency

R2: We apologize for the inconsistency and thank the reviewer for catching it. The gate in Eq. 8 is the intended formulation, the additional condition 1[At>0]\mathbb{1}[A_t > 0] in Algorithm 1 is redundant under our setting. Rewards are binary, and uninformative rollout groups in which all responses receive identical rewards are discarded during batch construction (Sec. 4.3 & Appendix B.1). Hence, within any valid group, a rollout is successful (Ri=1R_i = 1) if and only if its group-relative advantage is positive (AiA_i > 0). 1[success(o)]\mathbb{1}[success(o)] already implies 1[At>0]\mathbb{1}[A_t > 0]. Removing this redundant term therefore does not change the method’s behavior, and no negative-advantage token is ever amplified. We have removed 1[At>0]\mathbb{1}[A_t > 0] from Algorithm 1, so that now matches Eq. 8, and we have added a brief remark explaining why success gating already restricts amplification to positive-advantage tokens.

Revised sections: Appendix B.1 (Detail of TACT), Appendix G (Algorithm 1), Appendix H (pseudo-code).

W3: Incomplete ablation isolating IG's contribution.

R3: We have added ablations that isolate the contribution of the IG signal. All variants share TACT's trajectory filtering, gating and hyperparameters, and differ only in the multiplier applied to gated tokens:

  • w/o IG: trajectory filtering only, with no token-level multiplier.

  • Constant: every gated token receives the same multiplier, equal to the batch mean of MtM_t, which matches TACT's average amplification at every step.

  • Shuffled: the MtM_t values are randomly permuted across gated tokens within each batch. This preserves both the mean and the distribution of amplification, but removes the alignment between IG and individual tokens.

We draw three observations from these results:

Trajectory filtering alone does not explain the gains. The w/o IG variant reaches 39.97 Avg@16. Removing IG shaping will eliminate TACT's improvement.

IG alignment matters beyond amplification magnitude. The mean-matched constant multiplier recovers only a small part of the gain (40.39 Avg@16). The shuffled multiplier, which keeps the same mean and distribution of amplification, reaches 41.73. TACT still outperforms it by +0.78 Avg@16 and +4.10 Pass@16. Both controls receive the same average amplification as TACT, so the remaining gap comes from which tokens are amplified, not from a generic upweighting of positive examples.

The benefit is largest on the hardest benchmarks, such as Hmmt25 (+2.29 Avg@16, +3.33 Pass@16) and AIME25 (+1.04 / +3.33), where correct solutions depend more heavily on using intermediate tool results.

In summary, these results confirm the effectiveness of IG: the gains of TACT come from IG-guided, token-level credit assignment rather than from uniform advantage amplification.

Table R3(a). Pass@16 of IG ablations on Qwen2.5-7B. Best results are in bold. Δ denotes the difference between TACT and the best of the three variants.

MethodAIME24AIME25MATH500OlympiadAMC23Hmmt25MinervaAvg.
w/o IG56.6746.6795.0068.6092.5033.3339.7161.78
Constant70.0046.6794.8067.6092.5040.0038.2464.26
Shuffled56.6750.0097.0069.8092.5036.6739.3463.14
TACT70.0053.3395.4069.2097.5043.3341.9167.24
Δ+0.00+3.33−1.60−0.60+5.00+3.33+2.20+2.98

Table R3(b). Avg@16 of IG ablations on Qwen2.5-7B. Best results are in bold. Δ denotes the difference between TACT and the best of the three variants.

MethodAIME24AIME25MATH500OlympiadAMC23Hmmt25MinervaAvg.
w/o IG32.5021.2577.2842.9868.7515.2121.8039.97
Constant32.9222.2978.2244.8066.2516.0422.2440.39
Shuffled33.1223.9679.4646.4269.5316.4623.1641.73
TACT33.7525.0079.3346.6570.4718.7523.6442.51
Δ+0.63+1.04−0.13+0.23+0.94+2.29+0.48+0.78

Revised sections: Appendix D.7 (Ablation Analysis).

AuthorsOpenReview ↗

Response 3 for Reviewer yC9U

W4: Unexplained regressions and cross-family gap.

R4: We thank the reviewer for the careful reading and would like to clarify the following points.

1. We thank the reviewer for the careful reading. One correction first: on Qwen3-8B-Base, TACT outperforms SimpleTIR on OlympiadBench (56.66 vs. 55.75 Avg@16). The only regressions are on AMC23: −0.94 (4B) and −0.78 (8B). These gaps are small. AMC23 contains only 40 problems, and both methods are already near the ceiling on it. A gap of under 1 point therefore corresponds to only a handful of the 640 sampled responses and falls within run-to-run variance. Moreover, TACT achieves a higher Pass@16 on AMC23 for both models (100.00 vs. 97.50, +2.50), indicating that its performance upper bound is not reduced. We have added this analysis to Section 5.2.

2. Choice of base models. Following the standard setting in current tool-integrated agentic RL works [1–4], we adopt Qwen models. To cover both model generations and model sizes, we evaluate two families, Qwen2.5 and Qwen3, at three scales: 4B, 7B and 8B.

3.To further address this concern, we have trained TACT on the Llama-3.1-8B base model and compared it with SimpleTIR. As shown in Table R4(a) below, TACT outperforms SimpleTIR in both Avg@16 and Pass@16 on three benchmarks.

We find that the gains are largest on the more challenging competition benchmarks. On AIME24, TACT outperforms SimpleTIR in Pass@16, and on AMC23 it improves Pass@16 by 10.00 points. Moreover, as shown in Table R4(b), TACT reduces the average number of tool calls by 36.6% and the average code lines by 7.83, while still achieving higher accuracy. This is consistent with our observations on Qwen models and with the design of TACT: since credit is amplified only for post-tool tokens that actually draw on tool outputs in successful trajectories, redundant tool calls receive no additional reward. TACT consistently improves over SimpleTIR, indicating that tool-aware credit assignment is not specific to the Qwen family. In addition, the corresponding training curves are provided in Appendix D.9 of the revised paper.

Table R4(a). Results on Llama-3.1-8B base model. Best results are in bold. Δ denotes the difference between TACT and SimpleTIR.

MethodAIME24 Avg@16AIME24 Pass@16AMC23 Avg@16AMC23 Pass@16Minerva Avg@16Minerva Pass@16Avg. Avg@16Avg. Pass@16
base model0.213.330.9412.501.5912.870.919.57
SimpleTIR4.1713.3310.0030.006.9622.797.0422.04
TACT6.4626.6713.7540.008.0423.539.4230.07
Δ+2.29+13.34+3.75+10.00+1.08+0.74+2.38+8.03

Table R4(b). Average code lines and tool calls on Llama-3.1-8B base model.

AIME24AMC23MinervaAvg.
MethodCode LinesTool CallsCode LinesTool CallsCode LinesTool CallsCode LinesTool Calls
SimpleTIR84.353.9057.883.2128.272.3256.833.14
TACT64.692.1348.121.9634.201.8749.001.99
Δ−19.66−1.77−9.76−1.25+5.93−0.45−7.83−1.15

Related Revised Sections: Section 5.2 (Main Results), Appendix D.9 (Experiment on Llama Base Model)

[1] Feng J, Huang S, Qu X, et al. Retool: Reinforcement learning for strategic tool use in llms[C]//International Conference on Learning Representations. 2026.

[2]Li X, Zou H, Liu P. Torl: Scaling tool-integrated rl[J]. arXiv preprint arXiv:2503.23383, 2025.

[3]Xue Z, Zheng L, Liu Q, et al. Simpletir: End-to-end reinforcement learning for multi-turn tool-integrated reasoning[C]//International Conference on Learning Representations. 2026, 2026: 8424-8449.

[4]Feng L, Xue Z, Liu T, et al. Group-in-group policy optimization for llm agent training[J]. Advances in Neural Information Processing Systems, 2026, 38: 46375-46408.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely appreciate the thoughtful feedback and encouraging comments from all reviewers. Your insights have been invaluable in improving the clarity of our work and identifying key points for deeper analysis and future works. We have addressed each point in the individual rebuttals and also revised our main paper and appendix accordingly, with changes marked in red. Below, we summarize the common concerns with our responses and the corresponding modifications.

Comparison and Discussion on Related Works

  • Comparison with additional agentic RL baselines. We add ReTool and ToRL as baselines on the Qwen2.5-7B base model for detailed comparison. [W1-R1, Reviewer yC9U] and [Appendix D.1].
  • Relationship with InfoPO and novelty. We clarify the differences between TACT and InfoPO, namely to whom credit is assigned, how it enters the advantage, and on which trajectories it acts, and we state our contribution explicitly. [W2&Q2-R2, Reviewer NAqg] and [Sec. 4.2, Appendix D.10].

Clarification of Evaluation Results

  • Evaluation protocol for OlympiadBench. We add a detailed specification of the evaluation protocol. [Q3-R3, Reviewer NAqg] and [Appendix C.1].
  • Reliability of BFCL results. We re-evaluate both TACT and SimpleTIR three times on the BFCL-v4 Live and Non-Live splits and report mean ± std, providing a statistically reliable comparison. [Q4-R4, Reviewer NAqg] and [Sec. 5.3, Appendix D.8].
  • Per-benchmark regressions. We clarify that the only regressions are on AMC23, where TACT still achieves higher Pass@16. [W4-R4, Reviewer yC9U] and [Sec. 5.2].

Clarification of Algorithm

  • Consistency between Algorithm and Appendix H. We remove the redundant condition 1[At>0]\mathbb{1}[A_t > 0] from Algorithm 1 and explain why success gating already restricts amplification to positive-advantage tokens. [W2-R2, Reviewer yC9U] and [Appendix B.1, Algorithm 1].
  • Executed optimization and loss mask. We clarify that all results come from a single training procedure: trajectory-level filtering is performed during batch construction, and valid but incorrect rollouts keep their negative advantage. We correct two errors in the pseudo-code and separate validity from success explicitly. [W1&Q1-R1, Reviewer NAqg; W2&Q1-R2, Reviewer FG5Y] and [Sec. 4.3, Appendix B.1, Algorithm 1, Appendix H].

More Experimental Analysis of Ablation and Base Model

  • Isolating the contribution of IG. We add ablations that keep trajectory filtering and gating fixed and replace IG with no multiplier, a mean-matched constant multiplier, or a shuffled multiplier. These controls distinguish IG-guided attribution from generic upweighting of positive examples. [W3-R3, Reviewer yC9U; W1&Q2-R1, Reviewer FG5Y] and [Appendix D.7].
  • Performance in Llama base models. We further conduct training on Llama-3.1-8B base models for comparison. TACT improves over SimpleTIR in both Avg@16 (+2.38) and Pass@16 (+8.03), while reducing tool calls by 36.8%, demonstrating that its effectiveness is not specific to the Qwen family. [W4-R4, Reviewer yC9U] and [Appendix D.9].

We hope that our detailed responses have addressed the reviewers’ concerns and remain available for any follow-up questions. Thank you once again for your time and effort!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.