acceptodds
Under review as a conference paper at ICLR 2027

TIER: Fine-Grained Trajectory Intervention for Stable Agentic Reinforcement Learning

Abstract

Large language models can substantially improve their reasoning capabilities through multi-turn reinforcement learning with external tools, yet prolonged training often suffers from instability and performance collapse. A common stabilization strategy filters out trajectories exhibiting recognizable failures, making a binary decision: each trajectory is either discarded or allowed to contribute fully to the policy update. Such coarse intervention not only requires increasingly elaborate filtering criteria to balance stability against data retention, but also leaves the optimization influence of retained trajectories uncontrolled. We find that rollouts without obvious behavioral degeneration can produce unusually large gradients and exert disproportionate influence on the aggregate gradient. To address this issue, we introduce Trajectory Intervention via Eligibility and Rescaling (TIER), a fine-grained trajectory intervention framework. TIER uses permissive screening based on generic response-level statistics to remove clear degeneration without attempting to enumerate every possible failure mode. It then clips excessive gradient contributions from eligible trajectories before batch aggregation while preserving their update directions. This design enables three levels of intervention: discarding degenerate trajectories, attenuating overly influential ones, and fully retaining the remainder. On AIME24, TIER improves over the corresponding TIR baselines by 10.2 and 14.0 points on Qwen2.5-7B and Qwen3-4B-Base, respectively. Experiments in retrieval-based interaction further demonstrate TIER's effectiveness beyond mathematical tool use. Overall, our results show that stable agentic reinforcement learning benefits from controlling not only whether each trajectory contributes to learning, but also how strongly it does so.

Primary area:
reinforcement learning
Keywords:
Large Language Models, Agentic Reinforcement Learning, Tool-Integrated Reasoning, Training Stability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1tFTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors find that some coherent trajectories have huge gradient norms and dominate the batch update, and argue these should be scaled down rather than dropped. TIER filters degenerate trajectories with simple text statistics, then clips each remaining trajectory's gradient norm before aggregation. It is tested on math with two Qwen models and on Search-R1.

Critical strengths

The appendix is thorough, with ablations of each screening choice, threshold sweeps, and a clear account of the extra compute.

Critical weaknesses

The case for "attenuate, don't delete" is weak. The LLM audit says 82% of high-norm trajectories are useful, but there is no low-norm control group, so the number means little. The Hard-deletion collapse in Figure 4 may just come from removing a biased subset, for example mostly negative-advantage trajectories, not from losing useful signal. It is also a single run.

The paper never asks why some gradients are so large. One trajectory reaches a norm of 11,681 while typical norms are around 3 to 4. Without a diagnosis, or a comparison with simpler fixes like a tighter global clip, it is unclear that per-trajectory clipping is the right remedy.

The search results don't support transfer. TIER beats SimpleTIR by 1.3 points on average, but almost all of it comes from Bamboogle (52.0 vs 42.4). On the other six datasets the two are tied (46.17 vs 46.20), and Bamboogle is small and evaluated with one greedy rollout.

Questions for authors

I would consider raising my score if questions 1 and 2 show that high-norm trajectories really carry signal that random deletion does not remove, and question 4 shows a gain beyond Bamboogle.

  1. Please run the same LLM audit on a random sample of ordinary trajectories with norms below the cap and report their KEEP rate next to Table 7. If it is also around 82%, the audit says nothing specific about high-norm trajectories.

  2. In the Hard-deletion run, how many trajectories were deleted per step, and how were they split between positive and negative advantage? Please also add a control that deletes the same number of randomly chosen eligible trajectories per step. If the random version also collapses, the collapse comes from removing data, not from losing high-norm signal.

  3. What makes these gradient norms so large? Please break down the high-norm trajectories by response length, advantage sign, and the GSPO sequence-level importance ratio. Please also compare TIER with GSPO-TIR using a tighter global clip (for example 0.1 or 0.3) and the same eligibility screening.

  4. Please rerun the Search-R1 comparison between TIER and SimpleTIR with at least three seeds, and report results with and without Bamboogle.

  5. Some reported numbers are hard to reconcile with the evaluation setup. With Avg@16 on 30 problems, HMMT26 scores can only take certain values, and 13.4, 11.4, 3.6 and 23.9 in Table 1 are not among them. Please state how many HMMT26 problems are used and whether any were excluded. More generally, since the code is not yet available, could you share it during the discussion period?

11 comments

The authors did not provide any response during the rebuttal phase, so I am maintaining my score.

AuthorsOpenReview ↗

Response to Reviewer 1tFT (Part 1/4)

We apologize for the late reply, as there were quite a few experiments required for the rebuttal. We sincerely thank Reviewer 1tFT for recognizing the thorough appendix and for the concrete suggestions to strengthen our mechanistic evidence. We have carefully considered your concerns about the treatment of high-norm trajectories and added targeted controls and analyses to better explain the role of attenuation. We hope this evidence and clarification address your main concerns, as detailed in the responses below.


Q1: The case for "attenuate, don't delete" is weak. The LLM audit says 82% of high-norm trajectories are useful, but there is no low-norm control group, so the number means little. Please run the same LLM audit on a random sample of ordinary trajectories with norms below the cap and report their KEEP rate next to Table 7. If it is also around 82%, the audit says nothing specific about high-norm trajectories.

A1: Thank you for your helpful suggestion. Following it, we sampled 1,000 eligible ordinary low-norm trajectories from the same GSPO-TIR training phase and applied the same two blinded audit protocols used for the original high-norm cohort. The auditor was not shown gradient norms or group labels.

R3-Table1. Audit results for ordinary low-norm and high-norm trajectories.

CohortNBehavior KEEPUtility KEEPBoth KEEP
Random sample of eligible ordinary low-norm trajectories1,00098.70%91.20%89.90%
Original audited cohort: eligible trajectories with norm >601,16990.16%82.12%74.85%

The low-norm group has substantially higher behavior KEEP, utility KEEP, and joint KEEP rates. The LLM auditor therefore judges a larger fraction of ordinary low-norm trajectories to be worth retaining and of higher overall quality. This is consistent with the distinction we intend to make: high-norm trajectories contain more problematic behavior and have lower overall quality, yet they are not devoid of learning value. Removing all of them is therefore not an appropriate default.

Behavior KEEP indicates that a trajectory exhibits no clear behavioral degeneration. Utility KEEP additionally considers the trajectory's reward and asks whether it can provide a meaningful learning signal.

AuthorsOpenReview ↗

Response to Reviewer 1tFT (Part 2/4)

Q2: In the Hard-deletion run, how many trajectories were deleted per step, and how were they split between positive and negative advantage? Please also add a control that deletes the same number of randomly chosen eligible trajectories per step. If the random version also collapses, the collapse comes from removing data, not from losing high-norm signal.

A2: We sincerely appreciate your constructive feedback. We counted the trajectories deleted at each step in the original Hard-deletion run (totally 290 steps) and examined their advantage signs. Positive and negative deletion counts were similar over the first 120 steps. Positive-advantage trajectories became more prevalent thereafter, accounting for about 70% of all deletions in the later interval.

R3-Table2. Hard-deletion counts and advantage-sign composition over steps 1–290.

Training stepsTotal deletedPositive advantageNegative advantagePositive fractionNegative fraction
1–1202,8351,4361,39950.65%49.35%
121–2906,4324,5061,92670.06%29.94%
Total, 1–2909,2675,9423,32564.12%35.88%

The later deleted subset is indeed skewed toward positive-advantage trajectories. These counts alone do not determine whether training degradation produces more high-norm positive trajectories or whether removing more positive trajectories causes the collapse.

Following your recommendation, we trained two random-deletion controls independently from Qwen2.5-7B Base. Both follow the basic Hard-deletion setting and modify how trajectories are selected for deletion:

  1. Matching total deletion counts only. If Hard-deletion removes NN trajectories in an optimizer mini-batch at step tt, the random control uniformly samples NN trajectories without replacement from the corresponding batch's eligible candidates and zeros their gradient contributions. Candidates must pass the same eligibility checks, have valid loss tokens, and have finite, nonzero gradients. Sampling is not stratified by norm or advantage sign, so both ordinary- and high-norm trajectories may be selected. Counts are matched per mini-batch and therefore per step, while sign composition may differ.
  2. Matching positive and negative counts separately. If the reference mini-batch removes N+N_+ positive-advantage and N−N_- negative-advantage trajectories, we partition the eligible candidates by sign and uniformly sample the corresponding counts without replacement. This matches both the total deletion count and its sign composition.

Because Hard-deletion begins to collapse at around step 120 and is substantially degraded by step 200, we focus this comparison on roughly 200 training steps. Neither random control exhibits collapse as severe as Hard-deletion, although both perform worse than TIER.

R3-Table3. Online three-benchmark validation for Hard-deletion and the two random-deletion controls (Avg@16, %).

MethodMATH50AIME24AIME25Avg
Hard-deletion61.69.415.228.7
Random deletion: matched total count per mini-batch80.132.924.645.9
Random deletion: matched positive/negative counts per mini-batch73.529.618.540.5

These results support two observations:

  1. The Hard-deletion collapse is not explained solely by removing data or by a biased advantage-sign composition. We believe the removal of useful learning signals also matters. Random deletion may be less damaging because some selected low-norm trajectories, despite carrying useful signals, make relatively small gradient contributions.
  2. The sign composition of the removed data also affects performance: separately matching positive and negative counts performs worse than matching total counts alone.

Overall, the results suggest that data removal may contribute to collapse but is not the entire explanation. Losing useful contributions from high-norm trajectories is a plausible additional factor. Regardless of the precise cause, the Hard-deletion result cautions against treating high-norm trajectories in the same way as behaviorally ineligible trajectories and simply removing their contributions. This is the central purpose of the comparison.

Reviewer_aaW4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TIER, a trajectory-level intervention framework to improve training stability in multi-turn tool-integrated reinforcement learning. TIER combines two mechanisms: an eligibility controller that filters potentially degenerate trajectories using response-level statistics, and an influence controller that clips excessive per-trajectory gradients before batch aggregation. The authors argue that this approach can preserve useful learning signals while preventing individual trajectories from disproportionately influencing policy updates. Experiments on mathematical reasoning and search-based tool interaction demonstrate improved training stability and task performance compared with existing agentic RL baselines.

Critical strengths
  1. The paper demonstrates consistent empirical improvements across multiple settings. TIER achieves performance gains on mathematical reasoning benchmarks using Qwen2.5-7B and Qwen3-4B-Base, while additional experiments on search-based tool interaction further support its practical applicability.
  2. The experimental analysis is comprehensive. The authors provide extensive ablations of the eligibility and influence controllers, comparisons against hard deletion, threshold sensitivity studies, and trajectory-level gradient analyses. These experiments offer useful insights into the effects of different trajectory intervention strategies.
Critical weaknesses
  1. The methodological novelty of TIER is limited. The eligibility controller essentially replaces conventional binary trajectory filtering criteria with heuristic response-level statistics, while the influence controller applies the well-established technique of per-example gradient clipping at the trajectory level. Although their combination is practically effective, it is unclear what constitutes a substantial methodological contribution beyond combining these existing techniques.
  2. The specific contribution of the proposed eligibility controller is insufficiently isolated. Appendix Table 8 shows that simply adding trajectory-wise gradient clipping to the existing SimpleTIR filtering strategy achieves performance comparable to TIER on Qwen2.5-7B (44.2 vs. 44.0 average accuracy). While TIER performs better on Qwen3-4B-Base, Appendix Table 9 reveals that its eligibility controller filters substantially fewer trajectories than SimpleTIR. Therefore, it is unclear whether the improvement comes from the proposed ZR/EAD-based screening criteria or simply from retaining more trajectories during training.
Questions for authors
  1. Could the authors compare TIER against a simpler, more permissive filtering strategy with a matched trajectory retention rate and the same influence controller? This would help distinguish the benefits of the proposed response statistics from those of retaining more training trajectories.
  2. Since preventing training collapse is a primary motivation, could the authors report performance variance and collapse frequency across multiple independent training seeds?
  3. How sensitive are the manually selected ZR/EAD thresholds and gradient norm caps to changes in backbone models and tool environments? Can the same configuration be transferred without additional tuning?
7 comments
AuthorsOpenReview ↗

Response to Reviewer aaW4 (Part 1/3)

We appreciate Reviewer aaW4's recognition of our empirical results and analytical insights, and your suggestions for clarifying the contribution. We have made every effort to address your concerns about the sources of improvement and the strength of the evidence through targeted experiments, further analysis, and a clearer explanation of the method. We hope these additions help resolve your main concerns. Our point-by-point responses follow.


Q1: The methodological novelty of TIER is limited. The eligibility controller essentially replaces conventional binary trajectory filtering criteria with heuristic response-level statistics, while the influence controller applies the well-established technique of per-example gradient clipping at the trajectory level. Although their combination is practically effective, it is unclear what constitutes a substantial methodological contribution beyond combining these existing techniques.

A1: Thank you for your thoughtful feedback. We acknowledge that neither filtering nor gradient clipping is itself a new technique. However, TIER’s contribution goes beyond combining these existing techniques.

First, TIER introduces a different perspective on trajectory intervention. We argue that trajectory intervention should distinguish two fundamental questions: whether a trajectory should contribute to learning and how strongly it should contribute. Conventional binary filtering primarily addresses the former, leaving the potentially excessive influence of retained trajectories unaddressed.

Second, TIER translates this perspective into targeted algorithmic interventions. Eligibility control uses response-level statistics (ZR and EAD) to identify behavioral degeneration without requiring task-specific screening metrics. Influence control directly bounds trajectory-level gradients. While rewards, advantages, and importance ratios affect a trajectory’s contribution, the gradient is ultimately the quantity through which it influences the policy update.

Third, TIER provides empirical insights into how problematic trajectories should be treated. Behaviorally degenerate trajectories call for exclusion, whereas trajectories that carry useful learning signals but exhibit excessive gradients call for retention with attenuation. Our controller ablations and comparisons with same-threshold hard deletion and tighter global gradient clipping are designed to examine these distinctions.

Thus, TIER’s contribution lies not in introducing new primitive operations, but in identifying distinct mechanisms of training instability, developing targeted interventions, and systematically examining their roles in sustained RL training. We view the simplicity of the resulting algorithm as a strength rather than a limitation.

AuthorsOpenReview ↗

Response to Reviewer aaW4 (Part 2/3)

Q2: The specific contribution of the proposed eligibility controller is insufficiently isolated. Could the authors compare TIER against a simpler, more permissive filtering strategy with a matched trajectory retention rate and the same influence controller? This would help distinguish the benefits of the proposed response statistics from those of retaining more training trajectories.

A2: Thank you for this helpful suggestion. We have not found a suitable off-the-shelf method that simultaneously provides a simpler and more permissive screening rule. This illustrates a limitation of fixed heuristics such as SimpleTIR when adjusting filtering strictness, whereas TIER allows this strictness to be changed through its thresholds.

To examine the effect of filtering quantity, we constructed a more permissive version of SimpleTIR. Let MM be the number of trajectories filtered by the historical TIER run at a given step, and NN the number selected for filtering by SimpleTIR at the corresponding step. We use MM as an upper limit: if N<MN<M, we leave SimpleTIR's filtering decision unchanged, which occurs rarely in our experiment; if N>MN>M, we randomly select MM of those NN candidate trajectories for filtering.

We apply this strategy with the same influence controller as in the paper (Appendix C.2, Table 8) to train Qwen2.5-7B for 200 steps, and then compare the two settings under the same 200-step training budget. Within this shared training budget, the count-capped version performs worse than SimpleTIR with influence control and no count cap. Both checkpoints were evaluated on the same machine with the same complete six-benchmark Avg@16 protocol.

R2-Table1. Filtering-count control at fixed step 200 on Qwen2.5-7B (%, Avg@16).

SettingMATH500AIME24AIME25AIME26HMMT26OlympiadBenchAvg
Original SimpleTIR + influence control82.840.627.523.323.754.142.0
Count-capped SimpleTIR + influence control77.424.621.518.121.247.035.0

Original SimpleTIR has an advantage of 7.0 percentage points in the six-benchmark mean. Thus, simply filtering fewer trajectories and retaining more data does not necessarily improve training. Arbitrarily relaxing the filter may instead allow problematic trajectories to remain and harm performance. We believe TIER's gains over SimpleTIR arise from more precise exclusion of anomalous trajectories, with the additional retained trajectories carrying useful learning signals.

Q3: Since preventing training collapse is a primary motivation, could the authors report performance variance and collapse frequency across multiple independent training seeds?

A3: Thank you for your question. We added two mathematical TIER runs with different seeds under the same configuration, both covering 450 training steps. Time and resource constraints prevented us from running multiple GSPO-TIR seeds. Based on reports in related work, we expect the baseline collapse to be reproducible rather than resolved simply by changing the random seed. Including the run in the paper, we now have three independent TIER seeds:

R2-Table2. Three independent mathematical TIER runs on Qwen2.5-7B (%, Avg@16).

RunMATH500AIME24AIME25AIME26HMMT26OlympiadBenchAvg
Original run083.544.028.828.824.254.644.0
Rerun183.441.325.828.324.254.843.0
Rerun284.744.427.131.525.255.244.7
Mean ± sample SD83.9 ± 0.743.2 ± 1.727.2 ± 1.529.5 ± 1.724.6 ± 0.554.9 ± 0.343.9 ± 0.8

The mean across the three runs is 43.9%, with a sample standard deviation of 0.8 percentage points. We consider this variation modest and acceptable, supporting the consistency of TIER's stable-training performance across these runs.

AuthorsOpenReview ↗

Response to Reviewer aaW4 (Part 3/3)

Q4: How sensitive are the manually selected ZR/EAD thresholds and gradient norm caps to changes in backbone models and tool environments? Can the same configuration be transferred without additional tuning?

A4: We appreciate your valuable comments. Our theoretical considerations and empirical results suggest that eligibility intervals can be reused to some extent, while influence caps depend more strongly on gradient scale.

  1. ZR/EAD thresholds. These are relatively insensitive because the aim is to retain normal model responses, whose ZR/EAD ranges tend to be similar. In our mathematical experiments, the thresholds are largely shared across models, with small differences in some EAD boundaries. These differences were chosen by analyzing trajectories from lightweight pilot runs, as described in the paper, rather than through complete training runs for each candidate setting. Larger changes may be needed across environments because typical response structures differ. For example, nearly all interval settings were relaxed in the Search experiments.
  2. Gradient-norm caps. These are more sensitive because gradient scales differ substantially across models and environments. For example, Qwen3-4B-Base has smaller trajectory-gradient norms than Qwen2.5-7B in our mathematical experiments, and we accordingly use a smaller cap.

Overall, additional tuning is needed, but we consider the cost manageable. Approximately 10 initial baseline steps generally provide enough trajectories to guide our parameter choices.

Automatic threshold selection may further reduce this effort. Since the methods differ little in the earliest phase and GSPO-TIR also improves stably at that stage, one possibility is to run a few initial steps without intervention, estimate TIER's thresholds from the resulting trajectories, and then activate the controls.

Reviewer_FVPvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses training instability and collapse in multi-turn TIR. The authors observe that, besides clearly degenerate rollouts, behaviorally coherent trajectories can have abnormally large gradient norms that dominate the batch update. They propose TIER, which combines two components. An eligibility controller screens out degenerate trajectories using inexpensive response statistics at the turn and trajectory levels. An influence controller clips the gradient norm of each eligible trajectory before aggregation, preserving its direction. Experiments on tool-integrated mathematical reasoning with Qwen2.5-7B and Qwen3-4B-Base, plus a Search-R1-style retrieval setting, show that TIER stabilizes training and outperforms the GSPO-TIR and SimpleTIR baselines.

Critical strengths
  1. The method is simple, intuitive and well-motivated. The distinction between whether a trajectory contributes and how strongly it contributes is clearly articulated. The resulting three-level intervention is easy to understand.
  2. The improvements are clear and consistent. TIER outperforms both baselines on all six math benchmarks for both backbones.
  3. The empirical analysis is comprehensive. It includes controller ablations, threshold-sensitivity studies, ablations of the screening design, a rescaling-versus-hard-deletion comparison, and a transparent report of runtime and memory overhead.
Critical weaknesses
  1. The evaluation scope is narrow. All main experiments are on mathematical reasoning with a code interpreter, and the only other setting is a single Search-R1 QA experiment with one backbone. In that experiment, TIER‘s advantage over SimpleTIR (47.0 vs. 45.7) comes almost entirely from Bamboogle, which has only 125 questions. Excluding it, the averages over the other six datasets are essentially identical (46.2 vs. 46.2). Training collapse is most severe in long-horizon agentic settings, as documented by RAGEN and ARLArena. Evaluation on broader agentic benchmarks (e.g., ALFWorld, WebShop, multi-turn function calling or deep-search tasks) is therefore needed to support the claims of generality.
  2. The base models are outdated and small. Qwen2.5-7B and Qwen3-4B-Base were released well over a year ago, and no model larger than 7B is tested. More recent and stronger base models may produce far fewer degenerate trajectories and have different gradient-norm distributions, so it is unclear whether the gains persist, especially those from eligibility control. Results on recent base models and at larger scales are needed.
  3. The baselines are limited to GSPO-TIR and SimpleTIR. The paper does not compare against other stabilization methods discussed in its related work.
  4. The claim that the screening is task-agnostic is overstated. The thresholds are selected manually from preliminary runs and differ across backbones and settings. Code and final-answer turns need hand-crafted exemptions. This tuning cost limits the method’s practical applicability.
Questions for authors
  1. Could you please provide more experimental results on broader agentic benchmarks?
  2. Could you please evaluate your methods on more recent models such as Qwen 3.6/3.7/3.8?
  3. Could you please provide more results about other recent TIR methods?
2 comments
AuthorsOpenReview ↗

Response to Reviewer FVPv (Part 1/2)

We thank Reviewer FVPv for recognizing TIER's simple design and consistent gains in mathematical reasoning, as well as for the constructive suggestions regarding its broader applicability. Within the limited time and resources available during the rebuttal period, we have prioritized additional experiments on Qwen3.5-9B-Base and clarified the scope of our method. Our detailed responses follow.


Q1: The evaluation scope is narrow. Evaluation on broader agentic benchmarks (e.g., ALFWorld, WebShop, multi-turn function calling or deep-search tasks) is therefore needed to support the claims of generality. Could you please provide more experimental results on broader agentic benchmarks?

A1: We agree that evaluating TIER on a broader range of agentic benchmarks would be valuable. However, the requested experiments span several substantially different agentic environments and constitute a considerable extension of the current empirical study, which we believe goes beyond what can reasonably be accomplished within the rebuttal period.

At the algorithmic level, neither eligibility control nor influence control is specific to a particular task or tool environment. Our existing experiments on mathematical reasoning and search provide empirical evidence of TIER's applicability across distinct settings. We will clarify the scope of our empirical findings in the revised manuscript and leave a more comprehensive evaluation across agentic environments to future work.

Q2: In the search experiment, TIER‘s advantage over SimpleTIR (47.0 vs. 45.7) comes almost entirely from Bamboogle, which has only 125 questions. Excluding it, the averages over the other six datasets are essentially identical (46.2 vs. 46.2).

A2: Thank you for this careful analysis. We agree that TIER’s average improvement over SimpleTIR is primarily attributable to Bamboogle, with comparable performance on the remaining six datasets.

The primary purpose of the search experiments is to assess TIER’s applicability beyond mathematical tool use, rather than to establish its superiority across every datasets. Our experiments show that either eligibility control or influence control alone, even with relaxed thresholds, can maintain stable training in the current search setting. This suggests that search presents less severe stability challenges in our experiments, leaving less room for improvement. We therefore view these results as evidence of TIER’s broader applicability.

Q3: The base models are outdated and small. Could you please evaluate your methods on more recent models such as Qwen 3.6/3.7/3.8?

A3: Thank you for your question. The smallest open models in the Qwen3.6/3.7/3.8 series are Qwen3.6-27B and Qwen3.8-27B, both with 27B parameters. Given our time and resource constraints, we selected Qwen3.5-9B-Base for this extension. We trained the model with GSPO-TIR and TIER for 50 steps each, configuring TIER's two controls according to the parameter-selection procedure in the paper. We evaluated them on MATH500, AIME24, and AIME25. The two methods perform similarly on MATH500, while TIER scores substantially higher on the more challenging AIME24 and AIME25 benchmarks. R1-Table1 reports the results.

R1-Table1. Mathematical reasoning results on Qwen3.5-9B-Base at step 50 (%, Avg@16).

MethodMATH500AIME24AIME25Avg
GSPO-TIR98.167.160.675.3
TIER98.072.366.078.8

TIER scores higher on both AIME benchmarks and improves the equally weighted three-benchmark mean by 3.5 percentage points, demonstrating better average task performance under the same 50-step training budget.

We also examine code-based tool interaction in the training trajectories at step 50.

TIER also performs considerably better in code-based tool use: its valid code rate is 54.7%, compared with 22.5% for GSPO-TIR. This suggests that TIER has learned stronger code-generation capabilities.

AuthorsOpenReview ↗

Response to Reviewer FVPv (Part 2/2)

Q4: The baselines are limited to GSPO-TIR and SimpleTIR. The paper does not compare against other stabilization methods discussed in its related work. Could you please provide more results about other recent TIR methods?

A4: Thank you for your helpful suggestions.

We first experimented with the filtering rule from TAO-RL[1]. On Qwen2.5-7B, performance began to collapse after about 10 steps. We suspect that the rule's higher filtering rate, and the resulting reduction in effective training samples, contributed to this outcome.

We then collected results reported in recent papers, including TAO-RL[1], OTB[2], VCPO[3], ARPO[4], and GTPO[5]. ARPO and GTPO use Qwen2.5-7B-Instruct, a post-trained version of Qwen2.5-7B. The published results are presented below alongside TIER.

R1-Table2. Published results for recent mathematical TIR methods and TIER.

ModelAlgorithmMATH500AIME24AIME25AIME26HMMT26OlympiadBench
Qwen2.5-7BTAO-RL78.835.424.0——47.0
Qwen2.5-7BOTB84.741.528.1———
Qwen2.5-7BVCPO——27.8———
Qwen2.5-7B-InstructARPO78.830.030.0———
Qwen2.5-7B-InstructGTPO72.822.316.9———
Qwen2.5-7BTIER (ours)83.544.028.828.824.254.6
Qwen3-4B-BaseTAO-RL83.728.124.7——47.6
Qwen3-4B-BaseTIER (ours)85.341.530.232.726.358.1

These results were obtained with different training data and budgets and therefore do not constitute a controlled comparison. They nevertheless provide useful context: on the two backbones, TIER exceeds the reported results of the other methods on most benchmarks for which comparisons are available.

Q5: The claim that the screening is task-agnostic is overstated. The thresholds are selected manually from preliminary runs and differ across backbones and settings. Code and final-answer turns need hand-crafted exemptions. This tuning cost limits the method’s practical applicability.

A5: Thank you for your constructive feedback. We clarify that “task-agnostic” refers to the screening metrics rather than their thresholds. Compressibility and token diversity are applicable across tasks without requiring the design of new task-specific metrics. The thresholds are configurable, and output-format exemptions may depend on the task. We will clarify this distinction in the revised manuscript.

Importantly, threshold selection follows a unified tuning procedure across tasks, with relatively modest overhead: we typically run the baseline for a few steps (e.g., 10) and select the thresholds based on the collected trajectories.

[1] Tool-Aware Optimization with Entropy Guidance for Efficient Agentic Reinforcement Learning. Cao et al. arXiv:2606.03762, 2026.

[2] The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL. Li et al. arXiv:2602.07078, 2026.

[3] Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs. Huang et al. arXiv:2602.17616, 2026.

[4] Agentic Reinforced Policy Optimization. Dong et al. arXiv:2507.19849, 2025.

[5] Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization. Ding et al. ACL 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback. We have responded to each reviewer in detail and revised the manuscript, with all revisions marked in blue. We summarize our contributions and responses below.


Core Contributions

  1. A different intervention perspective. We distinguish whether a trajectory should contribute to learning from how strongly it should contribute, separating behavioral degeneration from excessive influence that binary filtering alone leaves unaddressed.
  2. Targeted algorithmic controls. Eligibility control identifies behavioral degeneration using response-level ZR and EAD statistics; influence control caps eligible trajectories' gradients, the quantities through which they affect policy updates.
  3. Empirical insights into trajectory treatment. Ablations and controlled comparisons support excluding degenerate trajectories but retaining and attenuating useful high-norm trajectories, rather than indiscriminately removing them.

New Experiments

  1. Low-norm audit control (1tFT-Q1). 1tFT noted that the LLM audit is uninformative without a low-norm reference. We conducted the same blinded LLM audit on 1,000 random eligible low-norm trajectories. The results demonstrate that high-norm trajectories are worse on average, yet most remain worth learning from, supporting attenuation over removal. We have added these results and the corresponding analysis to Appendix C.1.1 and Table 7 of the revised manuscript.
  2. Random-deletion controls (1tFT-Q2). 1tFT asked whether the Hard-deletion run collapses merely because less data remains. We tested two controls randomly deleting the same number of trajectories per mini-batch, matching the total or the positive/negative-advantage counts. Both degrade far less, so data loss alone does not explain the collapse.
  3. Retention-matched filtering (aaW4-Q2). aaW4 noted that TIER's gains over SimpleTIR may come from retaining more trajectories. We tested a control capping SimpleTIR's per-step filtered count at TIER's, but it performed worse than uncapped SimpleTIR under the same influence control: retaining more does not by itself help; precisely excluding the right trajectories matters.
  4. Tighter global clipping (1tFT-Q4). 1tFT asked whether a tighter global clip could replace per-trajectory control. Keeping eligibility screening, we disabled influence control and lowered the global clip from 1.0 to 0.5. This alleviates collapse, but performance plateaus late in training and remains below TIER's: uniformly shrinking the batch update is less effective than attenuating only the dominating trajectories.
  5. Independent runs (aaW4-Q3). aaW4 asked for variance and collapse frequency across seeds. We added two math TIER runs with the same configuration. The variance of all the three runs' scores on the six benchmarks is marginal, showing that TIER's stable training is reproducible across seeds.
  6. Newer backbone (FVPv-Q3). FVPv considered our backbones outdated and small, suggesting Qwen3.6+ models. Within our compute budget, we selected Qwen3.5-9B-Base for this extension. TIER exceeds GSPO-TIR both on evaluated results and code-generation capabilities, demonstrating its effectiveness on a newer, larger backbone.

Clarifications

  1. Purpose of Search (FVPv-Q2, 1tFT-Q5). Reviewers noted that TIER's Search gain over SimpleTIR comes mainly from the small Bamboogle set. We agree with that but we think search presents less severe stability challenges in our experiments, leaving less room for improvement. The primary purpose of the search experiments is to assess TIER’s applicability beyond mathematical tool use, rather than to establish its superiority across every datasets. We therefore view these results as evidence of TIER’s broader applicability, revising the abstract, introduction, Sec. 5.4 and App. D.2.
  2. Methodological contribution (aaW4-Q1). aaW4 viewed TIER as filtering plus per-example clipping. We agree these primitives are not new. Our contribution is separating whether a trajectory should be learned from and how strongly it should count, and showing that degenerate trajectories should be excluded but high-norm ones attenuated rather than deleted (Experiments 1–4).
  3. Threshold adaptation (FVPv-Q5, aaW4-Q4). Reviewers questioned the task-agnostic claim and whether thresholds transfer without tuning. "Task-agnostic" refers to the screening statistics, not their thresholds. As for the cost of additional threshold-tuning cost, we consider it manageable and acceptable.
  4. High-norm mechanisms (1tFT-Q3). 1tFT asked why some trajectory gradients become large. High-norm trajectories are shorter and have slightly lower GSPO sequence ratios, consistent with GSPO's length normalization and ratio–gradient coupling. As these factors interact, TIER bounds each trajectory's gradient directly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.