- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper examines response-level off-policy sequence masking in LLM reinforcement learning (Sections 3.1–3.2). The authors point out a structural flaw in standard geometric-mean sequence masking (GeoMean, Eqs. 4–5): because token log-ratios are averaged with their signs before taking the absolute value, large positive and negative token-level policy shifts within the same trajectory can cancel each other out. To prevent this cancellation, the paper introduces Cancellation-Aware Response Masking (CARM, Eqs. 6–7), which replaces the magnitude of the signed average with the average of the absolute token log-ratios, , and proves that thresholding deterministically bounds both the fraction and mean severity of sampled-token ratio-band violations (Proposition 3 and Corollary 1).
Empirically, CARM is plugged into asymmetric PPO/GRPO training (Eqs. 2–3, Algorithm 1) on mathematical reasoning (Qwen3.5-4B and Qwen3.5-9B trained on DAPO Math, evaluated on AIME 2024/2025/2026 and BeyondAIME in Table 1) and code generation (Qwen3.5-9B trained on TACO, evaluated on TACO test, LiveCodeBench-v6, HumanEval+, and MBPP+ in Table 2). Across these benchmarks, CARM outperforms Null, IcePop, TRM, and GeoMean, improving the four-dataset math average by up to +3.13 points over GeoMean on Qwen3.5-9B (70.42% vs. 67.29%) and the four-dataset code average by +2.88 points over the strongest baseline (IcePop, 68.31% vs. 65.43%). While the cancellation failure mode is real and the proposed fix is clean, the evaluation protocol—most notably selecting checkpoints directly on reported test benchmarks—and the lack of multi-seed variance reporting leave the empirical claims on shaky ground.
Critical strengths
1. Sharp, concrete diagnosis of bidirectional cancellation in geometric-mean sequence masking.
Rather than hand-waving that GeoMean can fail in theory, the paper pairs its toy counterexample (Proposition 1) with a compelling empirical dissection of an unfiltered 8,619-token rollout (Figure 1, Section 3.3, Appendix C) and a 1,024-response batch comparison (Figure 3). Showing that 195 out of 1,024 responses in a single batch are accepted by GeoMean () but rejected by CARM ()—and that the drift is spread across thousands of tokens rather than driven by a single spike (the largest token contributes only 0.43% of total )—convincingly motivates why signed sequence averages are blind to oscillatory drift.
2. A one-line drop-in modification with a clean token-level violation budget.
Replacing with in Algorithm 1 requires no extra forward passes or architectural changes, and Proposition 3 gives practitioners a direct way to interpret : for example, at the PPO clip band , setting guarantees that at most 8.88% of sampled tokens in any accepted response can fall outside the clip range (Section 3.4).
Critical weaknesses
1. Checkpoint and hyperparameter selection directly on the headline test benchmarks.
As noted under Soundness, Appendix D.3 (p. 26) and Appendix E (Table 4, p. 27) reveal that both the reported training checkpoint and the masking threshold (GeoMean swept over 5 values in 1.0025–1.03, CARM swept over 4 values in 1.02–1.04) were chosen to maximize three-year AIME mean@16 for math and LiveCodeBench-v6 pass@1 for code. Because AIME 2024/2025/2026 only have 30 problems each and RL checkpoints fluctuate noticeably across iterations, taking on the test metric itself can inflate gains and reward methods with higher iterate-to-iterate variance. A clean evaluation requires selecting checkpoints and on a held-out validation split (or reporting fixed-step final checkpoints) before touching AIME or LiveCodeBench-v6.
2. Point-estimate reporting despite three-seed runs, and uneven baseline tuning/controls.
Although Section 4.1 states that experiments use three random seeds, Tables 1 and 2 report only single numbers per cell without standard deviations, confidence intervals, or paired significance tests. Several differences are well within typical RL seed noise—for instance, on Qwen3.5-4B (Table 1), CARM and GeoMean tie on AIME 2025 (64.38%), CARM is slightly behind on AIME 2026 (69.79% vs. 70.62%), and the three-year AIME average differs by only 1.25 points (70.42% vs. 69.17%). Furthermore, while GeoMean and CARM had their thresholds swept on a grid (Table 3), TRM and IcePop were held at single fixed cutoffs (0.05 and ), and Section 5.2 (Figure 5) tests negative-advantage-only masking only by restricting CARM's own score rather than comparing against DeepSeek-V3.2's actual signed-log negative-advantage mask.
Questions for authors
-
Assessment-critical (Validation vs. test leakage in checkpoint/threshold selection): Can you report Table 1 and Table 2 results either at a fixed final training step across all methods, or using checkpoints and values selected on an independent validation set disjoint from AIME 2024–2026, BeyondAIME, and LiveCodeBench-v6? In particular, since
Qwen3.5-4BBeyondAIME already used a separate matched-iteration run (Appendix D.3, line 1387), how do the AIME and code numbers look under fixed-step evaluation? -
Assessment-critical (Multi-seed variance): Since you ran three seeds for both math and code (Section 4.1), please provide the 3-seed mean standard deviation (and/or paired bootstrap confidence intervals) for every row in Tables 1 and 2, as well as for the sensitivity table in Appendix E (Table 4).
-
Clarification (Baseline fairness & DeepSeek-style signed mask): In Section 5.1 (Figure 4),
GeoMean() andCARM() happen to mask almost the exact same fraction of sequences onQwen3.5-9B(52.75% vs. 52.32%), whereasTRMat its fixed 0.05 cutoff performs much worse on 4B (54.80%). What isTRM's rejection rate at 0.05, and how doesTRMperform when its cutoff is tuned to matchCARM's 52%–55% sequence rejection rate? Also, in Section 5.2, how doesCARMcompare against the original DeepSeek-V3.2 one-sided signed-log mask (rather than negative-onlyCARM)?
Thank you for highlighting both the rollout evidence for cancellation and the token-level guarantee for accepted responses.
1. How sensitive are the results to checkpoint selection?
We kept the thresholds and candidate checkpoints from Table 1 and selected each method’s checkpoint by its highest average mean@16 across AIME 2024, 2025, 2026, and BeyondAIME, weighting the four benchmarks equally.
This recovered all five published 9B checkpoints. On 4B, only Null, TRM, and GeoMean changed checkpoints, and the largest change in average accuracy was 0.86 percentage points. The ordering was unchanged at both sizes.
2. Mean and standard deviation
Thank you for pointing this out. We will add uncertainty estimates to the tables in the paper. Below is the Qwen3.5-9B code example, reported as mean pass@1 (%) ± standard deviation over three seeds.
3. What changes when TRM masks a comparable fraction of responses?
Thank you for raising this point. We initially used a TRM cutoff of 0.05, following the experimental setting in the original TRM paper [1]. At this cutoff, TRM masked 86.64% of responses on 4B and 71.76% on 9B, exceeding CARM’s corresponding rejection rates of 54.52% and 52.32%.
Motivated by your suggestion, we swept candidate cutoffs on 9B math to identify a setting with a comparable rejection rate, then conducted an additional training runs. With the threshold set at 0.3, the four-benchmark average improved from 66.92% to 69.52%. The corresponding cumulative sequence rejection rate was 54.11%, within the proposed 52–55% range.
Obtaining this control required an additional RL training run and substantial computation. We regret that our available compute budget limited the extent of additional training and prevented exhaustive threshold sweeps for every baseline. Your suggestion has strengthened the comparison by identifying a more effective TRM setting. The improvement illustrates that a cutoff that works in one training setting may need recalibration in another,and there may be no single threshold that works well for every model. It also shows that more aggressive filtering does not necessarily improve downstream accuracy.
4. How does CARM compare with the original DeepSeek-V3.2 mask?
In our early tests, two-sided GeoMean performed better than the original DeepSeek rule. As such, we chose GeoMean as the stronger baseline in our setting. Applying both GeoMean and CARM to responses of either advantage sign also let us compare how they aggregate token-level changes. We agree that DeepSeek’s choice to preserve positive-advantage responses merits a direct comparison.
We implemented Eq. 9 of DeepSeek-V3.2 [2]. It masks a response only when its advantage is negative and the mean signed log-ratio of rollout to current token probabilities exceeds the cutoff. Our math results and direct code comparison favor two-sided GeoMean and CARM in the tested settings. The comparisons can be found below:
If you are aware of an independent study comparing the original DeepSeek-V3.2 mask with two-sided GeoMean under comparable training and tuning budgets, please kindly let us know. We would be happy to discuss it in the revision.
We thank you again for the detailed feedback. We will clarify checkpoint selection, report uncertainty alongside the benchmark means, and include both the TRM comparison at similar rejection rates and the direct DeepSeek-V3.2 comparison in the revision. We hope you will reconsider your assessment of our paper.
References
[1] Li, Y., et al. Trust Region Masking for Long-Horizon LLM Reinforcement Learning. ICML 2026.
[2] DeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556, 2025.