- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes TACT, a modification of GRPO that combines trajectory filtering with token-level advantage reweighting. A second forward pass masks tool observations, and the resulting token log-likelihood difference determines how much to amplify successful post-tool reasoning. Experiments cover three Qwen checkpoints, seven mathematics benchmarks, and additional code-generation and function-calling evaluations. The authors report accuracy improvements, fewer tool calls, and modest additional training cost.
Critical strengths
The improvements are measured against SimpleTIR, not only against the unstable standard-TIR baseline. The reported mathematics average increases from 41.00 to 43.69 for the 4B model and from 40.21 to 42.51 for the 7B model. The reductions in tool calls are also useful, although they should not be equated with reductions in total generation cost.
Critical weaknesses
-
The evidence does not yet establish that the proposed signal improves tool-specific credit assignment rather than simply reweighting successful continuations. Equation (5) measures an observed token’s likelihood sensitivity to masked feedback; this need not indicate that the feedback helped solve the task. Appendix D.7 varies the gating scope, but does not isolate the information in the token weights from their average amplification. A constant or shuffled-weight control under identical filtering would address this central attribution issue.
-
The stated objective and pseudocode disagree about unsuccessful trajectories. Equation (9) and Appendix B retain the original advantages for valid but unsuccessful rollouts. Appendix H instead defines valid_mask = (seq_scores > 0) and multiplies the final loss mask by it, removing those rollouts from optimization altogether. This changes the learning objective, not merely the IG gate, and leaves unclear which method produced the reported results.
Questions for authors
-
Which loss mask was used for Tables 1–2? Please reconcile Equation (9), Appendix B, Algorithm 1, and Appendix H, and specify whether SimpleTIR used the same trajectory filtering.
-
With filtering and the success/post-tool gate fixed, does IG weighting outperform constant or shuffled multipliers matched for average amplification? This would distinguish the proposed attribution signal from generic positive-example reweighting.
8 comments
Response 2 for Reviewer FG5Y
W2&Q1: Method inconsistency
R2: We thank the reviewer and apologize for the confusion. All results in Tables 1, 2 come from the same training procedure, which matches Sec 4.3, Appendix B.1 and Algorithm 1:
-
Loss mask. The token-level loss mask excludes prompt tokens and tool-observation tokens. Valid but incorrect trajectories in retained groups keep their original group-relative advantage and still contribute negative-advantage updates.
-
Advantage shaping. Within valid trajectories, is applied only when the triple-gating condition $G_t \gets \mathbb{1}[\text{valid}(o)] \cdot \mathbb{1}[\text{success}(o)] \cdot \mathbb{1}[t \in \mathcal{I}_{\text{post}}] $ holds. Otherwise, the original advantage is retained (Eq. 9). Valid but incorrect rollouts therefore fall into the otherwise branch: they keep their original negative group-relative advantage and still contribute to the policy loss.
Reconciliation with Appendix H. The previous pseudo-code contained two errors: valid_mask = (seq_scores > 0) is in fact the success indicator, which is used only for the IG gate. final_loss_mask = loss_mask * valid_mask was erroneous and has been removed. We have rewritten Appendix H to separate validity from success explicitly. SimpleTIR uses its original void-turn filtering. For format-related filtering, TACT follows the same criterion as SimpleTIR.
Revised sections: Sec 4.3, Appendix B.1, Appendix C.1, Algorithm 1, Appendix H (Policy Optimization).
Thanks for the revisions and new experiments. The updated Appendix H resolves my concern about valid-but-wrong trajectories being excluded from the loss. The constant and shuffled ablations also directly answer my request and offer solid evidence for token-wise weighting. Two issues still stand.
- The implemented IG does not cleanly isolate tool feedback. As noted in Appendix B.1, factual log-probs are cached from the old policy, while counterfactuals are recomputed with the current actor before each mini-batch update. After the first update, their difference conflates policy drift and masked-observation effects — unlike the same-policy comparison in Eq. (5). The constant/shuffled baselines cannot tease these apart. Please quantify this mismatch, or run the comparison with both terms evaluated under a fixed policy. This is critical for interpreting the method as tool-aware credit assignment.
- The IG gains lack clear reliability. Tables 10–11 show +0.78 Avg@16 and +4.10 Pass@16 over shuffled weighting, with no mention of independent training repeats or uncertainty for these gaps. The three BFCL evaluation runs only cover evaluation noise, not training variance. Please share the seeds used for these ablations and paired per-seed results if you have them. The numbers are promising, but they do not yet prove the edge over random weighting is reproducible.
The revision is much better, and I am no longer worried about the loss masking issue. My main hesitation is whether the implemented signal and reported gains support the central credit-assignment claim. I will keep my score at 2.
Response 1 for Reviewer FG5Y
We thank the reviewer for the insightful and encouraging comments, which have been invaluable in improving our manuscript. We have addressed each weakness and question below, and revised the main paper and appendix accordingly, with all changes marked in red.
R1: We have added ablations that isolate the contribution of the IG signal. All variants share TACT's trajectory filtering, gating and hyperparameters, and differ only in the multiplier applied to gated tokens:
IG alignment matters beyond amplification magnitude. The mean-matched constant multiplier recovers only a small part of the gain (40.39 Avg@16). The shuffled multiplier, which keeps the same mean and distribution of amplification, reaches 41.73. TACT still outperforms it by +0.78 Avg@16 and +4.10 Pass@16. Both controls receive the same average amplification as TACT, so the remaining gap comes from which tokens are amplified, not from a generic upweighting of positive examples.
The benefit is largest on the hardest benchmarks, such as Hmmt25 (+2.29 Avg@16, +3.33 Pass@16) and AIME25 (+1.04 / +3.33), where correct solutions depend more heavily on using intermediate tool results.
In summary, these results confirm the effectiveness of IG: the gains of TACT come from IG-guided, token-level credit assignment rather than from uniform advantage amplification.
Table R1(a). Pass@16 of IG ablations on Qwen2.5-7B. Best results are in bold.
Table R1(b). Avg@16 of IG ablations on Qwen2.5-7B. Best results are in bold.
Revised sections: Appendix D.7 (Ablation Analysis).