acceptodds
Under review as a conference paper at ICLR 2027

CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learning

Abstract

Recent years have witnessed the rapid adoption of reinforcement learning (RL) in large language model (LLM) post-training, with substantial gains in mathematical reasoning and code generation. In practical systems, however, policy updates and differences between rollout and training engines can make sampled responses off-policy. Sequence-level masking addresses this mismatch by deciding whether an entire response should contribute to optimization. A common masking rule uses the length-normalized geometric mean of sampled token probability ratios. Its signed log-ratios can cancel across positions, concealing substantial bidirectional policy drift. We propose Cancellation-Aware Response Masking (CARM), a sequence-level mask that takes the absolute value of each token log-ratio before averaging, preventing opposing probability changes from canceling. We prove that accepted responses satisfy a joint bound on the fraction of sampled-token ratios outside a prescribed band and their mean log-distance beyond its boundaries. Experiments on mathematical reasoning and code generation show that CARM improves mean@16 averaged over AIME 2024/2025/2026 and BeyondAIME by up to percentage points over geometric-mean masking, and increases average pass@1 across four code benchmarks by points over the strongest evaluated baseline. These findings support CARM as a theoretically grounded and effective method for response-level off-policy control in LLM reinforcement learning.

Primary area:
reinforcement learning
Keywords:
Large Language Models, Reinforcement Learning, Off-Policy, Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rhQgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines response-level off-policy sequence masking in LLM reinforcement learning (Sections 3.1–3.2). The authors point out a structural flaw in standard geometric-mean sequence masking (GeoMean, Eqs. 4–5): because token log-ratios log⁡ri,t\log r _ {i,t} are averaged with their signs before taking the absolute value, large positive and negative token-level policy shifts within the same trajectory can cancel each other out. To prevent this cancellation, the paper introduces Cancellation-Aware Response Masking (CARM, Eqs. 6–7), which replaces the magnitude of the signed average with the average of the absolute token log-ratios, dCARM(yi)=1Ti∑t=1Ti∣log⁡ri,t∣d _ {\mathrm{CARM}}(y _ i) = \frac{1}{T _ i}\sum _ {t=1}^{T _ i} \lvert \log r _ {i,t} \rvert, and proves that thresholding sCARM(yi)=exp⁡(dCARM(yi))≤τs _ {\mathrm{CARM}}(y _ i) = \exp(d _ {\mathrm{CARM}}(y _ i)) \le \tau deterministically bounds both the fraction and mean severity of sampled-token ratio-band violations (Proposition 3 and Corollary 1).

Empirically, CARM is plugged into asymmetric PPO/GRPO training (Eqs. 2–3, Algorithm 1) on mathematical reasoning (Qwen3.5-4B and Qwen3.5-9B trained on DAPO Math, evaluated on AIME 2024/2025/2026 and BeyondAIME in Table 1) and code generation (Qwen3.5-9B trained on TACO, evaluated on TACO test, LiveCodeBench-v6, HumanEval+, and MBPP+ in Table 2). Across these benchmarks, CARM outperforms Null, IcePop, TRM, and GeoMean, improving the four-dataset math average by up to +3.13 points over GeoMean on Qwen3.5-9B (70.42% vs. 67.29%) and the four-dataset code average by +2.88 points over the strongest baseline (IcePop, 68.31% vs. 65.43%). While the cancellation failure mode is real and the proposed fix is clean, the evaluation protocol—most notably selecting checkpoints directly on reported test benchmarks—and the lack of multi-seed variance reporting leave the empirical claims on shaky ground.

Critical strengths

1. Sharp, concrete diagnosis of bidirectional cancellation in geometric-mean sequence masking.
Rather than hand-waving that GeoMean can fail in theory, the paper pairs its toy counterexample (Proposition 1) with a compelling empirical dissection of an unfiltered 8,619-token rollout (Figure 1, Section 3.3, Appendix C) and a 1,024-response batch comparison (Figure 3). Showing that 195 out of 1,024 responses in a single batch are accepted by GeoMean (τ=1.005\tau = 1.005) but rejected by CARM (τ=1.02\tau = 1.02)—and that the drift is spread across thousands of tokens rather than driven by a single spike (the largest token contributes only 0.43% of total ∣log⁡rt∣\lvert \log r _ t \rvert)—convincingly motivates why signed sequence averages are blind to oscillatory drift.

2. A one-line drop-in modification with a clean token-level violation budget.
Replacing ∣1Ti∑tlog⁡ri,t∣\lvert \frac{1}{T _ i}\sum _ t \log r _ {i,t} \rvert with 1Ti∑t∣log⁡ri,t∣\frac{1}{T _ i}\sum _ t \lvert \log r _ {i,t} \rvert in Algorithm 1 requires no extra forward passes or architectural changes, and Proposition 3 gives practitioners a direct way to interpret τCARM\tau _ {\mathrm{CARM}}: for example, at the PPO clip band [0.8,1.28][0.8, 1.28], setting τ=1.02\tau = 1.02 guarantees that at most 8.88% of sampled tokens in any accepted response can fall outside the clip range (Section 3.4).

Critical weaknesses

1. Checkpoint and hyperparameter selection directly on the headline test benchmarks.
As noted under Soundness, Appendix D.3 (p. 26) and Appendix E (Table 4, p. 27) reveal that both the reported training checkpoint and the masking threshold τ\tau (GeoMean swept over 5 values in 1.0025–1.03, CARM swept over 4 values in 1.02–1.04) were chosen to maximize three-year AIME mean@16 for math and LiveCodeBench-v6 pass@1 for code. Because AIME 2024/2025/2026 only have 30 problems each and RL checkpoints fluctuate noticeably across iterations, taking max⁡iter\max _ {\mathrm{iter}} on the test metric itself can inflate gains and reward methods with higher iterate-to-iterate variance. A clean evaluation requires selecting checkpoints and τ\tau on a held-out validation split (or reporting fixed-step final checkpoints) before touching AIME or LiveCodeBench-v6.

2. Point-estimate reporting despite three-seed runs, and uneven baseline tuning/controls.
Although Section 4.1 states that experiments use three random seeds, Tables 1 and 2 report only single numbers per cell without standard deviations, confidence intervals, or paired significance tests. Several differences are well within typical RL seed noise—for instance, on Qwen3.5-4B (Table 1), CARM and GeoMean tie on AIME 2025 (64.38%), CARM is slightly behind on AIME 2026 (69.79% vs. 70.62%), and the three-year AIME average differs by only 1.25 points (70.42% vs. 69.17%). Furthermore, while GeoMean and CARM had their thresholds swept on a grid (Table 3), TRM and IcePop were held at single fixed cutoffs (0.05 and [0.5,5.0][0.5, 5.0]), and Section 5.2 (Figure 5) tests negative-advantage-only masking only by restricting CARM's own score rather than comparing against DeepSeek-V3.2's actual signed-log negative-advantage mask.

Questions for authors
  1. Assessment-critical (Validation vs. test leakage in checkpoint/threshold selection): Can you report Table 1 and Table 2 results either at a fixed final training step across all methods, or using checkpoints and τ\tau values selected on an independent validation set disjoint from AIME 2024–2026, BeyondAIME, and LiveCodeBench-v6? In particular, since Qwen3.5-4B BeyondAIME already used a separate matched-iteration run (Appendix D.3, line 1387), how do the AIME and code numbers look under fixed-step evaluation?

  2. Assessment-critical (Multi-seed variance): Since you ran three seeds for both math and code (Section 4.1), please provide the 3-seed mean ±\pm standard deviation (and/or paired bootstrap confidence intervals) for every row in Tables 1 and 2, as well as for the τ\tau sensitivity table in Appendix E (Table 4).

  3. Clarification (Baseline fairness & DeepSeek-style signed mask): In Section 5.1 (Figure 4), GeoMean (τ=1.005\tau=1.005) and CARM (τ=1.025\tau=1.025) happen to mask almost the exact same fraction of sequences on Qwen3.5-9B (52.75% vs. 52.32%), whereas TRM at its fixed 0.05 cutoff performs much worse on 4B (54.80%). What is TRM's rejection rate at 0.05, and how does TRM perform when its cutoff is tuned to match CARM's 52%–55% sequence rejection rate? Also, in Section 5.2, how does CARM compare against the original DeepSeek-V3.2 one-sided signed-log mask (rather than negative-only CARM)?

1 comment

Thank you for highlighting both the rollout evidence for cancellation and the token-level guarantee for accepted responses.

1. How sensitive are the results to checkpoint selection?

We kept the thresholds and candidate checkpoints from Table 1 and selected each method’s checkpoint by its highest average mean@16 across AIME 2024, 2025, 2026, and BeyondAIME, weighting the four benchmarks equally.

Math, Average (%)NullIcePopTRMGeoMeanCARM
Qwen3.5-4B44.5536.9155.4162.0063.45
Qwen3.5-9B64.1357.9066.9267.2970.42

This recovered all five published 9B checkpoints. On 4B, only Null, TRM, and GeoMean changed checkpoints, and the largest change in average accuracy was 0.86 percentage points. The ordering was unchanged at both sizes.

2. Mean and standard deviation

Thank you for pointing this out. We will add uncertainty estimates to the tables in the paper. Below is the Qwen3.5-9B code example, reported as mean pass@1 (%) ± standard deviation over three seeds.

MethodTACOLiveCodeBench-v6HumanEval+MBPP+Average
Null34.50 ± 0.1038.46 ± 0.6585.57 ± 1.4175.22 ± 2.2558.44 ± 0.73
IcePop45.87 ± 0.4643.36 ± 0.9392.68 ± 2.7979.81 ± 0.5565.43 ± 0.83
TRM36.60 ± 1.3944.97 ± 0.6287.80 ± 1.6176.72 ± 0.5361.52 ± 0.70
GeoMean46.03 ± 0.9944.50 ± 0.5392.28 ± 1.2778.31 ± 0.0065.28 ± 0.36
CARM46.70 ± 0.5251.20 ± 1.0594.11 ± 0.7081.22 ± 1.2168.31 ± 0.74

3. What changes when TRM masks a comparable fraction of responses?

Thank you for raising this point. We initially used a TRM cutoff of 0.05, following the experimental setting in the original TRM paper [1]. At this cutoff, TRM masked 86.64% of responses on 4B and 71.76% on 9B, exceeding CARM’s corresponding rejection rates of 54.52% and 52.32%.

Motivated by your suggestion, we swept candidate cutoffs on 9B math to identify a setting with a comparable rejection rate, then conducted an additional training runs. With the threshold set at 0.3, the four-benchmark average improved from 66.92% to 69.52%. The corresponding cumulative sequence rejection rate was 54.11%, within the proposed 52–55% range.

Obtaining this control required an additional RL training run and substantial computation. We regret that our available compute budget limited the extent of additional training and prevented exhaustive threshold sweeps for every baseline. Your suggestion has strengthened the comparison by identifying a more effective TRM setting. The improvement illustrates that a cutoff that works in one training setting may need recalibration in another,and there may be no single threshold that works well for every model. It also shows that more aggressive filtering does not necessarily improve downstream accuracy.

4. How does CARM compare with the original DeepSeek-V3.2 mask?

In our early tests, two-sided GeoMean performed better than the original DeepSeek rule. As such, we chose GeoMean as the stronger baseline in our setting. Applying both GeoMean and CARM to responses of either advantage sign also let us compare how they aggregate token-level changes. We agree that DeepSeek’s choice to preserve positive-advantage responses merits a direct comparison.

We implemented Eq. 9 of DeepSeek-V3.2 [2]. It masks a response only when its advantage is negative and the mean signed log-ratio of rollout to current token probabilities exceeds the cutoff. Our math results and direct code comparison favor two-sided GeoMean and CARM in the tested settings. The comparisons can be found below:

EvaluationOriginal DeepSeek-V3.2 maskGeoMeanCARM
Math, mean@1658.07%67.29%70.42%
Code, pass@159.89%65.28%68.31%

If you are aware of an independent study comparing the original DeepSeek-V3.2 mask with two-sided GeoMean under comparable training and tuning budgets, please kindly let us know. We would be happy to discuss it in the revision.

We thank you again for the detailed feedback. We will clarify checkpoint selection, report uncertainty alongside the benchmark means, and include both the TRM comparison at similar rejection rates and the direct DeepSeek-V3.2 comparison in the revision. We hope you will reconsider your assessment of our paper.

References

[1] Li, Y., et al. Trust Region Masking for Long-Horizon LLM Reinforcement Learning. ICML 2026.

[2] DeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556, 2025.

Reviewer_Hv4LOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Cancellation-Aware Response Masking (CARM) for off-policy control in LLM reinforcement learning. CARM filters responses using the mean absolute token log probability ratio between the current and rollout policies, preventing opposing probability changes from canceling under geometric-mean aggregation. The authors derive bounds on the frequency and severity of sampled-token ratio violations and demonstrate improvements over several masking baselines on mathematical reasoning and code generation.

Critical strengths
  • The paper identifies a concrete limitation of signed log aggregation and illustrates it using actual rollout data. CARM requires only existing token log probabilities and integrates directly into the GRPO objective.
  • CARM improves aggregate performance across both mathematical reasoning and code generation. Comparisons at similar filtering rates provide evidence that the gains are not simply due to discarding more responses.
Critical weaknesses
  • The central modification replaces the absolute mean log ratio with the mean absolute log ratio. The theoretical bounds characterize this statistic but do not establish improved optimization or policy performance. Comparisons with other cancellation-free aggregation rules, such as root-mean-square log ratios, would better establish the value of the specific design. The theoretical analysis characterizes ratio violations conditional on a chosen threshold, but does not provide a principled rule for selecting a threshold that ensures stable or effective policy optimization. The reported results are sensitive to this choice: on Qwen3.5-9B, changing the threshold from 1.02 to 1.025 increases the AIME average from 72.99 to 77.50. It remains unclear how the threshold should be selected or transferred to new training settings without extensive tuning.
  • All experiments use Qwen3.5 models, with mathematical reasoning evaluated at 4B and 9B and code generation only at 9B. Since the distribution of token log-ratios may depend on the model and training configuration, these results do not establish whether CARM’s benefits or its threshold choices generalize across model families and larger scales. The GeoMean baseline is an author-defined variant rather than a direct implementation of DeepSeek-V3.2’s masking rule. DeepSeek combines a one-sided discrepancy criterion with negative advantage filtering, whereas CARM filters both positive and negative advantage responses based on unsigned drift. These choices reflect different optimization considerations, including whether to retain positive advantage learning signals despite policy mismatch. GeoMean is useful for isolating the aggregation change, but does not substitute for a direct comparison with the original method; the negative-only CARM ablation also leaves this gap unresolved.
Questions for authors

Do you have evidence that either the performance advantage or a fixed threshold transfers across model families or substantially different training configurations? How does CARM compare directly with the original DeepSeek-V3.2 rule, combining its one-sided discrepancy criterion with negative-advantage-only masking, under comparable tuning and training budgets?

1 comment

We thank the reviewer for highlighting the cancellation effect we documented in real rollouts. We regret, however, that the overall assessment does not fully recognize what the paper establishes beyond that observation. We prove that CARM bounds both the frequency and severity of sampled-token ratio violations in an accepted response, and show that addressing cancellation can improve downstream performance.

1. Do the performance gains and threshold choices transfer?

We tested this on Ministral-3-8B-Reasoning code generation, reusing the Qwen3.5-9B code thresholds of 1.02 for CARM and 1.01 for GeoMean. We selected each method’s checkpoint using the same LiveCodeBench-v6 criterion. The table reports pass@1 for each method.

MethodTACOLiveCodeBench-v6HumanEval+MBPP+Average
CARM28.10%53.23%90.24%72.40%60.99%
GeoMean28.20%50.76%72.56%61.20%53.18%

CARM gained 7.81 percentage points on average. TACO was effectively tied. The largest gains were on HumanEval+ and MBPP+, neither of which was used for this selection. We also selected each method’s checkpoint by its mean pass@1 across all four benchmarks, weighting the benchmarks equally. The highest observed means were 61.11% for CARM and 53.33% for GeoMean. The resulting 7.78-point gap was nearly identical to the 7.81-point gap under LiveCodeBench-based selection. These results show that the Qwen3.5-9B code thresholds transferred to a second model family on this task.

A single threshold that worked across all models and tasks would be desirable. However, in practice, even with identical weights and the same token prefix, the rollout engine and trainer can assign different log probabilities because they use different fused kernels, numerical precisions, or reduction orders [1, 2]. Policy updates between rollout and training introduce additional drift. The same cutoff can therefore filter different fractions of responses across setups.

What transfers exactly is the interpretation of the threshold. With the paper’s PPO ratio interval of [0.8, 1.28], a CARM threshold of 1.02 guarantees that at most 8.88% of sampled tokens in any accepted response fall outside the interval. This upper bound on the fraction tightens as the average severity of the violations increases. The theory thus gives practitioners a way to specify a tolerance for drift. Its effect on downstream accuracy still requires validation.

We agree that other cancellation-free statistics are plausible. Our experiments include TRM, which applies a cutoff to half the largest squared token log-ratio in a response. We chose the mean absolute log-ratio because several moderate shifts can contribute even in the absence of an extreme token, and the score is straightforward to compute from the log probabilities of the response alone. We also examined your RMS suggestion on 5,760 archived responses. At matched retention, RMS and CARM shared 2,740 of the 2,756 responses each retained.

2. How does CARM compare with the original DeepSeek-V3.2 masking rule?

In our early tests, two-sided GeoMean performed better than the original DeepSeek rule. As such, we chose GeoMean as the stronger baseline in our setting. Applying both GeoMean and CARM to responses of either advantage sign also let us compare how they aggregate token-level changes. We agree that DeepSeek’s choice to preserve positive-advantage responses merits a direct comparison.

We implemented Eq. 9 of DeepSeek-V3.2 [2]. It masks a response only when its advantage is negative and the mean signed log-ratio of rollout to current token probabilities exceeds the cutoff. Our Qwen3.5-9B math results and direct code comparison favor two-sided GeoMean and CARM in the tested settings. The comparisons can be found below:

EvaluationOriginal DeepSeek-V3.2 maskGeoMeanCARM
Math, mean@1658.07%67.29%70.42%
Code, pass@159.89%65.28%68.31%

If you are aware of an independent study comparing the original DeepSeek-V3.2 mask with two-sided GeoMean under comparable training and tuning budgets, please kindly let us know. We would be happy to discuss it in the revision.

We thank the reviewer again for the detailed feedback. We will clarify what the threshold guarantees, report the cross-family results, and include the original DeepSeek comparison. We hope you will reconsider your assessment in light of the theoretical contribution and the empirical evidence.

References

[1] Zhong, T., et al. Diagnosing Training Inference Mismatch in LLM Reinforcement Learning via a Zero-Mismatch Reference. arXiv:2605.14220, 2026.

[2] Qi, P., et al. Defeating the Training-Inference Mismatch via FP16. arXiv:2510.26788, 2025.

[3] DeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556, 2025.

Reviewer_F5jxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies response-level masking for off-policy LLM RL. The authors observe that geometric-mean masking averages signed token log-ratios, allowing positive and negative policy shifts to cancel. They propose CARM, which instead averages absolute token log-ratios and rejects responses with large aggregate deviation.

The paper derives bounds relating the CARM threshold to the frequency and severity of token-level ratio violations. Experiments on mathematical reasoning and code generation show that CARM generally outperforms geometric-mean masking, TRM, and other evaluated baselines.

Critical strengths
  1. The cancellation issue is clearly identified and the proposed fix is simple and intuitive. CARM directly measures the magnitude of bidirectional policy drift rather than allowing opposing token-level changes to cancel.

  2. The empirical evaluation is reasonably strong. The experiments cover math and code generation, multiple model sizes, three random seeds, and several masking baselines. CARM shows fairly consistent improvements, especially on the larger model and code benchmarks.

Critical weaknesses
  1. The methodological novelty is limited. CARM essentially replaces the absolute value of the mean log-ratio with the mean absolute log-ratio. While this avoids cancellation, the modification is very simple and follows directly from the identified issue.

  2. The theoretical contribution is also fairly limited. The main properties largely follow from elementary inequalities once the mean absolute log-ratio is chosen. For example, the acceptance-set inclusion follows immediately from the triangle inequality. I do not think the current theory substantially deepens our understanding of off-policy RL beyond justifying the proposed masking statistic.

  3. It is unclear whether cancellation itself is the fundamental quantity that should determine whether a response is useful for optimization. Large bidirectional token-level changes do not necessarily imply that the resulting policy update is harmful. The empirical results support CARM, but the paper provides limited analysis connecting its masking criterion to optimization bias, variance, or actual off-policy error.

Questions for authors
  1. Beyond preventing cancellation, is there a theoretical reason why mean absolute log-ratio should be the appropriate response-level measure of off-policy error? Can it be connected directly to optimization bias or error in the policy-gradient estimate?

  2. How does CARM compare with other simple non-canceling statistics, such as RMS log-ratio, quantile-based filtering, or the fraction of tokens outside the PPO ratio interval? This seems important for establishing that the particular CARM statistic is necessary rather than simply one of many reasonable alternatives.

  3. Since CARM and GeoMean can produce different rejection rates at the same threshold, can the authors provide a more controlled comparison where thresholds are chosen to exactly match the average retained-response rate?

1 comment

Thank you for recognizing the cancellation issue and for asking the harder question of when it matters for learning. We agree that the change is simple and that acceptance-set inclusion follows from the triangle inequality. We should have explained more clearly why this small change matters in practice. In the 8,619-token response in Fig. 1, the absolute mean log-ratio was 0.00250, while the mean absolute log-ratio was 0.02449. The largest token contributed only 0.43% of the total absolute drift. GeoMean accepted it at 1.005 and CARM rejected it at 1.02. The discrepancy arose across many token decisions and could be detected using log probabilities the trainer already computes.

1. Connection to policy-gradient error.

There is a limited theoretical connection at a fixed prefix. Let qq and pp be the rollout and current next-token distributions and let r(a)=p(a)/q(a)r(a)=p(a)/q(a). Since 1−r≤−log⁡r1-r\leq-\log r for 0<r≤10<r\leq1, TV(p,q)=E[max⁡(1−r(a),0)]≤E[∣log⁡r(a)∣].\mathrm{TV}(p,q) = \mathbb{E}[\max(1-r(a),0)] \leq \mathbb{E}[|\log r(a)|].

Thus, for a fixed advantage-weighted score function h(a)h(a) with ∥h(a)∥≤B\|h(a)\|\leq B, changing the action-sampling distribution at that prefix changes its mean by at most 2B⋅Eq∣log⁡r∣2B\cdot\mathbb E_q|\log r|. This gives the absolute log-ratio an interpretation beyond preventing cancellation. It also identifies the limit of the argument. A rollout observes only one action at each visited prefix, and masking changes which prefixes enter the update. The bias and variance of the full masked PPO gradient remain open. What we can guarantee for an individual accepted response is a violation budget. At τ=1.02\tau=1.02, at most 8.88% of its sampled-token ratios can fall outside the PPO band [0.8,1.28][0.8,1.28], with a tighter bound when violations extend farther beyond the boundaries.

2. RMS, quantiles, and violation counts.

Mean absolute log-ratio charges each unit of log-distance equally in either direction. RMS puts more weight on large deviations and also implies our violation bound at the same log cutoff. A quantile leaves the magnitude of the tail above it unrestricted, while a violation count gives equal weight to a token just outside the band and one far beyond it.

To compare their actual decisions, we replayed all four statistics on 5,760 archived responses from a Qwen3.5-9B CARM math run. We applied each rule to the same pre-mask token records and chose a cutoff for each to retain exactly 2,756 responses, the number retained by CARM at τ=1.025\tau=1.025.

Alternative ruleResponses retained in common with CARM, out of 2,756
RMS log-ratio2,740
95th percentile of absolute log-ratio2,747
Fraction of tokens outside the PPO band2,736

We thank the reviewer for suggesting these alternatives. When we set each threshold to retain the same number of responses, each alternative retained more than 99% of the responses selected by CARM. Thus, on this sample, the choice among these four statistics made little difference to which responses were kept. One reason we use the mean absolute log-ratio is that its threshold has a direct interpretation: for each accepted response, it bounds the fraction of tokens whose absolute log-ratio can exceed a given value. We agree that these alternatives and their close agreement with CARM deserve discussion in the paper.

3. Exactly matched retention versus GeoMean.

On the same 5,760 responses, a single GeoMean threshold of 1.011209 retained exactly 2,756 responses, matching CARM's 47.85% retention. The two rules kept 2,714 in common and each kept 42 responses the other discarded. On a separate unfiltered 4B batch, the difference was larger. Both rules kept 161 of 1,024 responses, but only 131 were shared. These controls show how much the identity of the retained data can change at exactly equal counts.

For downstream accuracy, Figure 4 shows what happened when we trained each method with its original threshold. On 9B, CARM masked 52.32% of responses over steps 1–120 versus 52.75% for GeoMean, and achieved 77.50% versus 74.03% on AIME mean@16 averaged over 2024–2026. On 4B, CARM likewise masked fewer responses (54.52% versus 55.65%) and achieved higher accuracy (70.42% versus 69.17%). Thus, on both model sizes, CARM’s higher accuracy came with a slightly lower fraction of responses masked. This directly addresses the concern that CARM gained accuracy simply by discarding more training responses.

Overall, we thank the reviewer again for the thoughtful and constructive feedback. We will revise the paper to present our contribution more clearly and better explain the scope of our findings. We hope you will reconsider your assessment of our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.