- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors find that some coherent trajectories have huge gradient norms and dominate the batch update, and argue these should be scaled down rather than dropped. TIER filters degenerate trajectories with simple text statistics, then clips each remaining trajectory's gradient norm before aggregation. It is tested on math with two Qwen models and on Search-R1.
Critical strengths
The appendix is thorough, with ablations of each screening choice, threshold sweeps, and a clear account of the extra compute.
Critical weaknesses
The case for "attenuate, don't delete" is weak. The LLM audit says 82% of high-norm trajectories are useful, but there is no low-norm control group, so the number means little. The Hard-deletion collapse in Figure 4 may just come from removing a biased subset, for example mostly negative-advantage trajectories, not from losing useful signal. It is also a single run.
The paper never asks why some gradients are so large. One trajectory reaches a norm of 11,681 while typical norms are around 3 to 4. Without a diagnosis, or a comparison with simpler fixes like a tighter global clip, it is unclear that per-trajectory clipping is the right remedy.
The search results don't support transfer. TIER beats SimpleTIR by 1.3 points on average, but almost all of it comes from Bamboogle (52.0 vs 42.4). On the other six datasets the two are tied (46.17 vs 46.20), and Bamboogle is small and evaluated with one greedy rollout.
Questions for authors
I would consider raising my score if questions 1 and 2 show that high-norm trajectories really carry signal that random deletion does not remove, and question 4 shows a gain beyond Bamboogle.
-
Please run the same LLM audit on a random sample of ordinary trajectories with norms below the cap and report their KEEP rate next to Table 7. If it is also around 82%, the audit says nothing specific about high-norm trajectories.
-
In the Hard-deletion run, how many trajectories were deleted per step, and how were they split between positive and negative advantage? Please also add a control that deletes the same number of randomly chosen eligible trajectories per step. If the random version also collapses, the collapse comes from removing data, not from losing high-norm signal.
-
What makes these gradient norms so large? Please break down the high-norm trajectories by response length, advantage sign, and the GSPO sequence-level importance ratio. Please also compare TIER with GSPO-TIR using a tighter global clip (for example 0.1 or 0.3) and the same eligibility screening.
-
Please rerun the Search-R1 comparison between TIER and SimpleTIR with at least three seeds, and report results with and without Bamboogle.
-
Some reported numbers are hard to reconcile with the evaluation setup. With Avg@16 on 30 problems, HMMT26 scores can only take certain values, and 13.4, 11.4, 3.6 and 23.9 in Table 1 are not among them. Please state how many HMMT26 problems are used and whether any were excluded. More generally, since the code is not yet available, could you share it during the discussion period?
11 comments
Response to Reviewer 1tFT (Part 1/4)
We apologize for the late reply, as there were quite a few experiments required for the rebuttal. We sincerely thank Reviewer 1tFT for recognizing the thorough appendix and for the concrete suggestions to strengthen our mechanistic evidence. We have carefully considered your concerns about the treatment of high-norm trajectories and added targeted controls and analyses to better explain the role of attenuation. We hope this evidence and clarification address your main concerns, as detailed in the responses below.
Q1: The case for "attenuate, don't delete" is weak. The LLM audit says 82% of high-norm trajectories are useful, but there is no low-norm control group, so the number means little. Please run the same LLM audit on a random sample of ordinary trajectories with norms below the cap and report their KEEP rate next to Table 7. If it is also around 82%, the audit says nothing specific about high-norm trajectories.
A1: Thank you for your helpful suggestion. Following it, we sampled 1,000 eligible ordinary low-norm trajectories from the same GSPO-TIR training phase and applied the same two blinded audit protocols used for the original high-norm cohort. The auditor was not shown gradient norms or group labels.
R3-Table1. Audit results for ordinary low-norm and high-norm trajectories.
| Cohort | N | Behavior KEEP | Utility KEEP | Both KEEP |
|---|---|---|---|---|
| Random sample of eligible ordinary low-norm trajectories | 1,000 | 98.70% | 91.20% | 89.90% |
| Original audited cohort: eligible trajectories with norm >60 | 1,169 | 90.16% | 82.12% | 74.85% |
The low-norm group has substantially higher behavior KEEP, utility KEEP, and joint KEEP rates. The LLM auditor therefore judges a larger fraction of ordinary low-norm trajectories to be worth retaining and of higher overall quality. This is consistent with the distinction we intend to make: high-norm trajectories contain more problematic behavior and have lower overall quality, yet they are not devoid of learning value. Removing all of them is therefore not an appropriate default.
Behavior KEEP indicates that a trajectory exhibits no clear behavioral degeneration. Utility KEEP additionally considers the trajectory's reward and asks whether it can provide a meaningful learning signal.
Response to Reviewer 1tFT (Part 2/4)
Q2: In the Hard-deletion run, how many trajectories were deleted per step, and how were they split between positive and negative advantage? Please also add a control that deletes the same number of randomly chosen eligible trajectories per step. If the random version also collapses, the collapse comes from removing data, not from losing high-norm signal.
A2: We sincerely appreciate your constructive feedback. We counted the trajectories deleted at each step in the original Hard-deletion run (totally 290 steps) and examined their advantage signs. Positive and negative deletion counts were similar over the first 120 steps. Positive-advantage trajectories became more prevalent thereafter, accounting for about 70% of all deletions in the later interval.
R3-Table2. Hard-deletion counts and advantage-sign composition over steps 1–290.
| Training steps | Total deleted | Positive advantage | Negative advantage | Positive fraction | Negative fraction |
|---|---|---|---|---|---|
| 1–120 | 2,835 | 1,436 | 1,399 | 50.65% | 49.35% |
| 121–290 | 6,432 | 4,506 | 1,926 | 70.06% | 29.94% |
| Total, 1–290 | 9,267 | 5,942 | 3,325 | 64.12% | 35.88% |
The later deleted subset is indeed skewed toward positive-advantage trajectories. These counts alone do not determine whether training degradation produces more high-norm positive trajectories or whether removing more positive trajectories causes the collapse.
Following your recommendation, we trained two random-deletion controls independently from Qwen2.5-7B Base. Both follow the basic Hard-deletion setting and modify how trajectories are selected for deletion:
- Matching total deletion counts only. If Hard-deletion removes trajectories in an optimizer mini-batch at step , the random control uniformly samples trajectories without replacement from the corresponding batch's eligible candidates and zeros their gradient contributions. Candidates must pass the same eligibility checks, have valid loss tokens, and have finite, nonzero gradients. Sampling is not stratified by norm or advantage sign, so both ordinary- and high-norm trajectories may be selected. Counts are matched per mini-batch and therefore per step, while sign composition may differ.
- Matching positive and negative counts separately. If the reference mini-batch removes positive-advantage and negative-advantage trajectories, we partition the eligible candidates by sign and uniformly sample the corresponding counts without replacement. This matches both the total deletion count and its sign composition.
Because Hard-deletion begins to collapse at around step 120 and is substantially degraded by step 200, we focus this comparison on roughly 200 training steps. Neither random control exhibits collapse as severe as Hard-deletion, although both perform worse than TIER.
R3-Table3. Online three-benchmark validation for Hard-deletion and the two random-deletion controls (Avg@16, %).
| Method | MATH50 | AIME24 | AIME25 | Avg |
|---|---|---|---|---|
| Hard-deletion | 61.6 | 9.4 | 15.2 | 28.7 |
| Random deletion: matched total count per mini-batch | 80.1 | 32.9 | 24.6 | 45.9 |
| Random deletion: matched positive/negative counts per mini-batch | 73.5 | 29.6 | 18.5 | 40.5 |
These results support two observations:
- The Hard-deletion collapse is not explained solely by removing data or by a biased advantage-sign composition. We believe the removal of useful learning signals also matters. Random deletion may be less damaging because some selected low-norm trajectories, despite carrying useful signals, make relatively small gradient contributions.
- The sign composition of the removed data also affects performance: separately matching positive and negative counts performs worse than matching total counts alone.
Overall, the results suggest that data removal may contribute to collapse but is not the entire explanation. Losing useful contributions from high-norm trajectories is a plausible additional factor. Regardless of the precise cause, the Hard-deletion result cautions against treating high-norm trajectories in the same way as behaviorally ineligible trajectories and simply removing their contributions. This is the central purpose of the comparison.
The authors did not provide any response during the rebuttal phase, so I am maintaining my score.