acceptodds
Under review as a conference paper at ICLR 2027

Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination

Abstract

Test items from public benchmarks leak into pretraining corpora, and once memorized they would inflate evaluation scores. Contamination mitigation evaluation intervenes in decoding to suppress memorization and restore a contaminated model's genuine capability. However, its prevailing metric, the G-AP (Gap of Aggregate Performance), has three flaws. First, a discrete correct/incorrect readout cannot characterize per-question performance. Second, averaging before differencing lets over- and under-suppression cancel. Third, uniform per-question weighting rewards restoring the clean model's most frequent solve probabilities while neglecting the rarer ones, which differ in how hard they are to restore. To address these issues, we propose SA-PPG (Stratified Aggregate of Per-question Probability Gaps), which estimates each question's solve probability by sampling, differences it against the clean model per question, and aggregates the resulting gaps within groups defined by the clean model's solve probability. Further, on the mitigation side, since existing strategies first estimate where contamination lies and then act on that estimate, they are only as accurate as the estimate. Our strategy, RailCap, instead judges contamination during generation: whenever a sample falls back onto the greedy trajectory, the next trajectory token is capped to the runner-up, and suppression accumulates until the response distribution is sufficiently dispersed. Extensive experiments shows that SA-PPG excel G-AP in estimating the restoration of prior strategies, and RailCap attains the lowest SA-PPG in all nine settings (three models three contamination domains). Code is available at https://anonymous.4open.science/r/railcap.

Primary area:
datasets and benchmarks
Keywords:
benchmark contamination, data contamination, evaluation metrics, contamination mitigation, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6ZS5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to evaluate and improve contamination mitigation, where a decoding-time intervention is supposed to bring a contaminated model’s benchmark performance back to that of a clean reference model. On the metric side, it argues that the prevailing G-AP metric has three flaws. It relies on a noisy single-sample 0/1 readout, it averages before differencing so that over- and under-suppression cancel, and it weights questions uniformly, which lets a trivial All-Zero strategy look competitive when the clean model is weak. The authors build up G-APP, A-PPG, and finally SA-PPG, which uses sampled solve probabilities, takes absolute per-question gaps against the clean model, and averages these gaps equally across bins of the clean model’s solve probability. On the method side, the paper proposes RailCap. RailCap indexes the contaminated model’s greedy trajectory by n-grams and, whenever a sample re-enters that trajectory, caps the logit of the next trajectory token at the runner-up value. Experiments use LoRA-simulated contamination on Llama-2-7B, Gemma-4-E2B, and Pythia-12B across GSM8K, a paraphrased GSM8K variant (PQ), and HumanEval. Under SA-PPG, RailCap is best in all nine settings, and the ranking of prior methods changes substantially relative to G-AP, most notably for LNE-blocking.

Critical strengths
  • A precise and useful diagnosis of the prevailing metric. The sign decomposition shows cleanly that G-APP ≤ A-PPG and that a zero aggregate gap can come entirely from cancellation. Table 1 makes this concrete, since LNE-blocking’s near-zero G-AP hides an over-suppression component twice as large as RailCap’s. Figure 2 adds solid evidence that single-sample 0/1 readouts are not reproducible even for the same model. Several published mitigation papers report G-AP as their headline number.
  • A simple mitigation with consistent results. RailCap needs no auxiliary detector, no neuron localization, and no access to the clean model. It costs one greedy decode and an n-gram lookup per question. It outperforms all baselines across three models and three domains, and the observation that the entropy-based LNE-blocking falls below Identity on PQ is a valuable finding in its own right.
Critical weaknesses
  • SA-PPG has a loophole of the same kind it was designed to close, and its validation is circular. Equal weighting across bins implicitly assumes a uniform prior over clean solve probabilities, so a constant strategy that sets every solve probability to 0.5 scores about 0.25. On Llama-2 × GSM8K this beats Identity, TED, and LNE-blocking while ignoring the questions entirely. SA-PPG thus replaces the All-Zero loophole with a collapse-to-the-middle one. Its support also rests largely on RailCap winning under it, which is circular without an independent test against known ground truth.

  • RailCap’s claim of needing no contamination estimate is overstated. RailCap treats the greedy trajectory as the contamination signal and applies the cap to every question. On unleaked questions where the model is confidently correct, samples also follow the greedy path, so RailCap likely suppresses genuine capability there, which is the failure mode the paper attributes to estimate-based methods. Without Δ⁺ and Δ⁻ split by leaked and unleaked questions, it remains unclear whether RailCap removes memorization or simply injects diversity

Questions for authors
  • What SA-PPG does a constant strategy r ≡ c attain for c in [0, 1] in each setting? If c = 0.5 outperforms most prior methods, how do you reconcile this with the argument that SA-PPG fixes the All-Zero loophole of A-PPG?
  • Can you report Δ⁺ and Δ⁻ separately for leaked and unleaked questions for RailCap and each baseline? In particular, how much does RailCap reduce solve probability on unleaked questions relative to Identity?
  • The absolute per-question gap is biased upward under finite sampling, and Figure 2 already shows a clean-versus-clean gap of about 0.041 at m = 50. What is the SA-PPG of the clean model against itself in each of the nine settings, and are RailCap’s margins over the runner-up larger than this floor and than variation across seeds?
  • Equal bin weighting gives a bin with one or two questions the same weight as a bin with hundreds. How many questions fall in each bin, and how sensitive are the rankings to coarser or equal-frequency binning? Are the bins defined using the same clean-model samples that enter the gaps?
Reviewer_xykHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies evaluation and mitigation of benchmark contamination in LLMs. It proposes SA-PPG, a metric that uses per-question solve probabilities, computes gaps before aggregation, and stratifies questions by clean-model difficulty, as well as RailCap, a decoding-time mitigation method that detects when sampled generations fall back onto the contaminated model’s greedy trajectory and suppresses the corresponding next token toward the runner-up. Experiments show that SA-PPG changes the ranking of existing mitigation methods compared with G-AP, while RailCap achieves the lowest SA-PPG across all nine model–domain settings.

Critical strengths
  1. The paper is clearly written and well organized, with a smooth progression from identifying problems in existing methods, to proposing the new metric and mitigation approach, and finally to experimental validation.
Critical weaknesses
  1. SA-PPG: (1) My main concern is that the contribution feels very incremental. I agree that the three issues identified with existing metrics are real to some extent, but each of them seems relatively minor and the proposed fixes are also quite straightforward. Overall, I did not find the metric contribution particularly novel or exciting. (2) More importantly, I find the experimental validation somewhat circular. As I understand it, the paper does not construct an independent ground truth for mitigation quality (for example, through controlled settings with known degrees of contamination or known degrees of restoration). Instead, the experiments largely instantiate the same three failure modes used to motivate SA-PPG and then show that SA-PPG behaves as intended on them. This demonstrates that the metric is consistent with the authors’ motivation, but does not convincingly establish that it is a more faithful measure of true mitigation quality.

  2. RailCap: The method feels highly heuristic. I understand that it is motivated by the observations in Figure 1, but these observations are obtained from a relatively limited set of settings and are then directly turned into a decoding rule. It is unclear whether the same pattern holds more broadly across models, benchmarks, and contamination mechanisms, and the paper provides little theoretical justification for why this particular intervention should be generally effective. Honestly, I do not understand why the trajectory token should specifically be capped at the runner-up probability.

Questions for authors
  1. Could the authors report how often RailCap actually intervenes in practice (e.g., the average number of caps per response)?

  2. What is the justification for assigning equal weight to all non-empty solve-probability strata in SA-PPG? In particular, why should a stratum containing very few questions receive the same weight as one containing many questions? Is there a principled or ground-truth-based justification for this weighting choice, and how sensitive are the conclusions to it?

4 comments

We thank the reviewer for the careful review. We address the concerns below.

Reply for W1:

Our validation is not circular.

  • Cancellation makes G-AP wrong, not merely imprecise, and we prove this analytically rather than test it empirically (§3.3.2). A gap of averages equals ∣Δ+−Δ−∣|\Delta^+-\Delta^-|, so it can be zero even when no question is restored, whereas A-PPG and SA-PPG are zero if and only if every question is restored.
  • The readout and weighting fixes are practical patches for this setting, each supported by experiments. The probability readout is more stable (Fig. 2). Stratification is necessary because contamination inflates performance, so this setting tends to favor strategies that uniformly degrade the model: All-Zero beats TED and Shortcut under uniformly weighted A-PPG, yet ranks last under SA-PPG (Table 1).

Our contribution is not incremental.

  • Correcting G-AP raises LNE-blocking's gap 12.5-fold, from near-perfect (0.0235) to barely better than no mitigation (0.2932 vs. 0.3261). It also changes the ranking of prior strategies: LNE-blocking, the best under G-AP, falls behind Shortcut (0.2932 vs. 0.2476; Table 1).
  • Correcting this error helps the field develop genuinely effective mitigation strategies, rather than ones that only look effective under G-AP.

Reply for W2:

The runner-up pattern (Fig. 1b) holds across models, contamination types, and benchmarks. The table gives the share of forks at which the clean model's token is the contaminated model's runner-up, at the k-th fork of each leaked question. From the third fork on, this share is 52–63% on GSM8K and PQ and above half on HumanEval.

Setting1st3rd5th7th9th
Llama-2 × GSM8K43%58%57%56%59%
Gemma × GSM8K41%55%58%56%61%
Pythia × GSM8K41%58%58%56%54%
Llama-2 × PQ43%61%58%59%63%
Gemma × PQ39%57%57%56%58%
Pythia × PQ45%52%58%57%60%
Llama-2 × HumanEval27%69%70%56%92%
Gemma × HumanEval26%59%53%71%83%
Pythia × HumanEval33%67%65%65%59%

Each component of RailCap reduces one error component. We justify each design choice from an evolutionary perspective: starting from LNE-blocking, we swap in RailCap's components one at a time and track which error component each reduces (Llama-2 × GSM8K):

StrategySuppressionTriggerΔ⁺ (under)Δ⁻ (over)G-APPSA-PPG
LNE-blockingblockentropy estimate0.10490.08310.02180.2913
LNE-capcapentropy estimate0.13140.04560.08580.1990
RailCapcaprail check0.11580.04560.07020.1917
  • Cap (Fig. 1b): where the two models agree, the clean model's token is the contaminated model's top token; where they fork, it is most often the runner-up. Capping keeps both candidates, whereas blocking discards the top token, so over-suppression nearly halves.
  • Rail check (Fig. 1a): leaked samples collapse onto the greedy trajectory, so each fall-back onto it flags memorization during generation. Suppressing at these points lowers under-suppression without adding over-suppression.
  • Diagnosis needs per-question gaps: G-APP rates the cap about 4× worse, because the cap removes the over-suppression that was cancelling LNE-blocking's under-suppression. Judged by a gap of averages, the cap would have been discarded.

Reply for Q1:

RailCap intervenes sparsely, and more on leaked questions. Average number of caps per response, in the setting of Table 2:

SettingLeakedUnleaked
Llama-2 × GSM8K8.765.96
Gemma × GSM8K5.063.80
Pythia × GSM8K4.373.52
Llama-2 × PQ6.355.06
Gemma × PQ3.873.30
Pythia × PQ3.643.19
Llama-2 × HumanEval5.113.41
Gemma × HumanEval4.873.34
Pythia × HumanEval3.423.14
  • Sparse: RailCap caps 3–9 tokens per response on average; on Llama-2 this is 5–9 caps per 100 generated tokens, so over 90% of decoding steps sample from the contaminated model's unmodified distribution.
  • Adaptive: leaked responses receive more caps than unleaked ones in all nine settings (1.1–1.5×) at nearly identical response lengths, without any up-front contamination estimate: how much each response is suppressed is decided online by how often it falls back onto the greedy trajectory (Fig. 1a).

Reply for Q2:

Equal weighting requires a strategy to restore every difficulty level. A mitigation strategy should restore questions at every difficulty level (stratum of the clean model's solve probability), rather than fit the most common one. On Llama-2 × GSM8K, most questions are hard for the clean model: on 53%, its solve probability is below 0.1. Under uniform per-question weighting, All-Zero, which fails every question, is therefore close to the clean model on most questions and beats TED and Shortcut (A-PPG; Table 1). Equal stratum weights mirror balanced accuracy in imbalanced classification, which averages accuracy over classes rather than samples so that a majority-class classifier cannot score well.

The conclusions do not depend on the number of strata B. We use B = 50 because its stratum width, 1/50, matches the finest resolution of a solve probability estimated from m = 50 samples: each stratum holds a single attainable value (only 49/50 and 1 share the last one). The table varies B from 1 (a single stratum, i.e., A-PPG) to 50 on Llama-2 × GSM8K:

StrategyB=1B=5B=10B=15B=20B=25B=30B=35B=40B=45B=50
Identity0.37930.32030.32490.32550.32580.32720.32800.32430.32730.32540.3261
LNE-blocking0.18790.28250.28880.29590.29570.29310.29540.29380.29600.29790.2932
Shortcut0.23990.23980.24450.24690.24610.24740.24880.24640.24530.24690.2476
RailCap0.16340.18910.19140.19240.19350.19290.19380.19260.19500.19520.1914
All-Zero0.21900.48120.48860.49590.49490.49040.49680.49890.49770.49910.4903
Smallest stratum (questions)1319803621141366665

For every B from 2 to 50, the ranking from best to worst is the same: RailCap, Shortcut, LNE-blocking, Identity, All-Zero. It changes only from B = 1 to B = 2, i.e., when stratification is introduced: All-Zero drops from third to last, behind even Identity (no mitigation), and LNE-blocking falls behind Shortcut. The ranking already holds at B = 5, where every stratum has at least 80 questions, so it does not rely on sparse strata.

And the revised paper adds Figure 3, which computes SA-PPG for every B from 1 to 50 in all nine settings: the curves flatten out well before B = 50, and RailCap is the lowest one across essentially the whole range, so the comparison does not depend on how B is chosen.

We thank the reviewer again for the constructive comments. These additional analyses and the per-question protocol will be incorporated into the revised manuscript. We would be grateful for the reviewer's feedback on these clarifications, and we welcome any further suggestions.

Thank you for the additional experiments and for the careful rebuttal. I read the response closely, and I understand the authors’ arguments and the new empirical results.

That said, my overall assessment remains largely unchanged. I still find the proposed fixes quite incremental. I agree that the issues identified in prior metrics are real, but the corresponding modifications are very straightforward and contain relatively little problem-specific insight; similar changes could be applied to many evaluation metrics.

I also remain unconvinced by the validation of SA-PPG. Fundamentally, there is still no independent criterion for evaluating the final metric beyond the arguments and desiderata used to motivate it. For example, one could construct settings where, for two mitigation methods A and B, an external ground truth independently establishes that A achieves better restoration than B, and then test whether SA-PPG recovers this ordering more reliably than G-AP. Without such a criterion, the evidence mainly shows that SA-PPG behaves consistently with the authors’ intended notion of restoration, rather than independently establishing that it is a more faithful measure of restoration quality.

Similarly, the additional experiments for RailCap do not substantially change my concern that the method is highly empirical and heuristic.

I also appreciate the additional analysis for Q2. However, this does not address my more fundamental concern about the lack of an independent ground-truth criterion for the metric. Without such an independent criterion, I do not see how the central validity question of the proposed metric can be resolved by further analyses of its own desired properties or robustness.

I welcome any further discussion.

Thanks for your careful review and additional reply.

A strategy family with a known ordering. Starting from Identity, we force a failure (solve probability 0) on a share jj of HH, the unleaked questions that M_cl solves with probability at least 0.5. Questions are destroyed in a fixed random order, so a larger jj destroys every question a smaller jj destroys, and nothing else changes.

  • A larger jj is necessarily a worse strategy. A destroyed question's gap becomes rcl(q)r_{cl}(q), which is at least its previous gap ∣rs(q)−rcl(q)∣|r_s(q)-r_{cl}(q)| because rcl(q)≥0.5r_{cl}(q)\ge 0.5. No question moves closer to M_cl, and the destroyed ones move farther. Since these questions are unleaked, destroying them removes no memorization, only genuine capability.
  • Only SA-PPG rises reliably on every model. G-AP and G-APP fall as jj grows on all four models: they reward destroying capability, because the added over-suppression cancels the inflation on leaked questions (OLMo-2-32B: G-AP 0.2153 → 0.0015). A-PPG rises, but on Pythia-12B, whose clean model is weak, by only 0.0018 per row, which is comparable to its sampling noise. Across 1,000 simulated evaluations that redraw 50 samples per question, A-PPG increases strictly over all six rows in 40% of them and SA-PPG in 98%. Both reach 100% on the other three models; G-AP and G-APP reach 0% everywhere.

Rows give jj; j=0j=0 is Identity, and for Llama-2-7B this row matches Identity in Table 1.

Pythia-12B (clean accuracy 0.074, |H| = 25)

Destroyed share of HG-APG-APPA-PPGSA-PPG
0.00.13340.13480.15910.2131
0.20.13100.13280.16090.2413
0.40.12860.13080.16270.2696
0.60.12610.12880.16450.2978
0.80.12370.12680.16630.3255
1.00.12130.12480.16810.3538

Llama-2-7B (clean accuracy 0.219, |H| = 116)

Destroyed share of HG-APG-APPA-PPGSA-PPG
0.00.31920.32430.37930.3261
0.20.30790.31300.38860.3492
0.40.29660.30170.39790.3724
0.60.28480.28990.40750.3966
0.80.27350.27860.41680.4198
1.00.26230.26730.42610.4431

Gemma-4-E2B (clean accuracy 0.227, |H| = 123)

Destroyed share of HG-APG-APPA-PPGSA-PPG
0.00.25400.23220.30940.3147
0.20.24240.22090.31870.3375
0.40.23130.21000.32750.3594
0.60.21980.19880.33660.3820
0.80.20870.18800.34540.4039
1.00.19710.17660.35470.4268

OLMo-2-32B (clean accuracy 0.539, |H| = 362)

Destroyed share of HG-APG-APPA-PPGSA-PPG
0.00.21530.22030.31210.3606
0.20.17290.17790.35030.3868
0.40.12970.13490.38920.4133
0.60.08710.09240.42760.4395
0.80.04410.04950.46630.4661
1.00.00150.00700.50470.4924
Reviewer_x26COpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to evaluate and improve contamination mitigation, i.e., strategies that intervene in the decoding of a contaminated LLM to suppress memorization and bring its behavior closer to that of a clean model. The authors argue that the standard G-AP metric has three flaws: a single 0/1 readout per question is a noisy estimate of per-question performance; averaging before differencing lets under- and over-suppression cancel out; and uniform per-question weighting allows a trivial strategy that simply degrades the model to obtain a good score when the clean model is weak. They instead propose SA-PPG, which estimates each question's solve probability from multiple samples, computes the gap to the clean model per question, and averages the absolute gaps within groups defined by the clean model's solve probability and then across groups. The authors also propose RailCap, a mitigation strategy that, whenever a sample falls back onto the contaminated model's greedy trajectory, caps the logit of the next trajectory token at the second-largest logit. Contamination is thus judged step by step during generation rather than estimated up front, as in prior strategies. In experiments with three models (Llama-2, Gemma-4, Pythia) and three contamination domains (GSM8K, a paraphrased version of GSM8K, and HumanEval), SA-PPG substantially changes the ranking of prior strategies relative to G-AP on Llama-2 × GSM8K, the only setting where both metrics are reported, and RailCap achieves the lowest SA-PPG in all nine model × domain settings.

Critical strengths

Metric (SA-PPG). It is shown that the standard G-AP metric is misleading: formally and experimentally, it is evident that a zero gap is possible without restoration. When the metric is changed, the conclusions about previous methods are reversed: by G-AP, LNE-blocking looks almost ideal, whereas by SA-PPG it is only slightly better than Identity. SA-PPG consistently corrects all three defects, and the contribution of each correction was verified separately. Stratification by difficulty does not allow a strategy to obtain a good score simply through degradation of the model.

Method (RailCap). RailCap is simple and practical: it requires only the logits of the contaminated model, with no training, no reference model, and no training data. The design relies on a measured observation (the clean model's token is often second in rank for the contaminated model) and is confirmed by ablation. RailCap yields the lowest SA-PPG in all 9 model × domain combinations, including under contamination with paraphrased questions. All baselines were run under a unified protocol, and the code is released.

Critical weaknesses

Metric (SA-PPG). The validity of SA-PPG is not checked against an external criterion, and the underlying goal is not justified: the paper does not explain why restoring each individual question matters more than restoring the aggregate benchmark score. The metric is not validated against any independent measure, such as strategies with a known degree of restoration, performance on fresh, distributionally equivalent questions (e.g., GSM1k, GSM-Plus), or the ranking of models with different degrees of contamination. Moreover, some SA-PPG groups contain only a few questions yet receive the same weight as large ones, which may make the metric noisy; the number of groups and the equal weighting are not ablated.

Method (RailCap). The comparison with baselines may be biased. To estimate solve probability, all baselines are switched to sampling at T = 0.7, although LNE-blocking was designed for greedy decoding: its original evaluation is deterministic, so the readout-noise argument does not apply to it. TED's sampling procedure and Shortcut's patching source were also modified (Appendix C). It is also unclear whether RailCap generalizes. The method treats falling back onto the greedy trajectory as a sign of memorization, but a strong model is also confident on problems it can solve on its own. The paper does not test RailCap on an uncontaminated model (i.e., whether it degrades clean performance), uses a single contamination regime (5 LoRA epochs), and covers only relatively small, weak base models and two task types (math and code). Finally, no confidence intervals or multi-seed results are reported, although the gaps between methods are sometimes small.

Questions for authors

Could affect the evaluation

  1. Why is the goal to restore each individual question rather than the aggregate benchmark score? If a strategy produces the correct aggregate by compensating for errors, why is that bad in practice? Can SA-PPG be validated against an external criterion: strategies with a known degree of restoration, quality on fresh equivalent questions (GSM1k, GSM-Plus), or ranking of models with different degrees of contamination?
  2. How does RailCap affect the uncontaminated model M_cl? Does it degrade its quality on questions that it solves confidently? Can results be reported separately for leaked and non-leaked questions?
  3. Does RailCap's advantage hold if it is compared with LNE-blocking under the original greedy protocol? And does the ranking of methods remain the same?
  4. What is the spread of SA-PPG across seeds? Are the differences between methods significant? How many questions are in the smallest groups, and how does SA-PPG change with a different number of groups or with weights that depend on group size?

Clarifying questions

  1. How does SA-PPG relate to the fidelity and resistance metrics from Sun et al. (ICML 2025), the latter of which likewise compares the contaminated model with the clean one on a per-question basis? How does RailCap differ from MemFree (Ippolito et al., 2023), which also intervenes in decoding by n-grams? Why is there no comparison with DeconIEP (2026)?
  2. How was n = 4 chosen? The sweep in Table 3 was done on Llama-2 × GSM8K, that is, on one of the test configurations and using SA-PPG itself. Was n tuned on held-out data, and how much do the results in the other eight configurations depend on it?
4 comments

(Part 1/3)

We thank Reviewer x26C for the careful and thorough review, and for the detailed suggestions that led to the additional analyses summarized below.

Reply for Q1:

Why per-question rather than aggregate restoration? We believe per-question restoration is the right target for two reasons:

  1. It is stricter and implies aggregate restoration. SA-PPG(s) = 0 if and only if every question is restored, which implies a zero aggregate gap; the converse fails (§3.3).
  2. It does not depend on test-set composition. A per-question gap cannot be offset by other questions, whereas a zero aggregate gap reached through compensation holds only for the exact mix it was measured on; in practice, which questions are leaked varies across models and is unknown.

External validation of SA-PPG. We appreciate the suggestions, but in our view none of them can validate the metric independently:

  • Known-restoration strategies: assigning a "known" degree of restoration already requires a restoration metric, which makes the check circular.
  • Models with different degrees of contamination: these only provide different settings; a model's degree of contamination does not tell us how much a strategy restores on it.
  • Fresh questions: GSM1k has no per-question correspondence with GSM8K. GSM-Plus perturbs the GSM8K test questions themselves, so leakage carries over. We ran Llama-2 × GSM8K on it: M_co exceeds M_cl by +6.0 points on leaked questions vs. +0.9 on unleaked ones. Used as a "strategy" (M_co answering the GSM-Plus), GSM-Plus attains a G-AP of 0.041, close to the best strategies, while its SA-PPG of 0.262 lies within the range of existing strategies.

Instead, we ground SA-PPG analytically: we prove that SA-PPG(s) = 0 if and only if the mitigated model matches the clean model on every question (§3.3–3.4), a guarantee that G-AP lacks.

Reply for Q2:

Interference with genuine capability. A mitigation strategy should leave the model's behavior unchanged on questions it has not memorized. On Llama-2 × GSM8K, we apply each strategy to M_cl and to the unleaked questions of M_co, and report SA-PPG against M_cl as in Table 2.

StrategyM_cl, leakedM_cl, unleakedM_co, unleaked
TED0.0850.0850.198
RailCap0.1570.1570.180
LNE-blocking0.2970.2920.285

Applied to M_cl, RailCap does interfere to some extent, though about half as much as LNE-blocking. On the unleaked questions of M_co, which every metric in Table 2 includes, RailCap attains the lowest SA-PPG.

Stronger models. SA-PPG on GSM8K:

StrategyLlama-3.1-70BOLMo-2-32B
No mitigation0.3840.361
TED0.3670.361
LNE-blocking0.3250.294
RailCap0.2570.225

RailCap remains the most effective strategy on both stronger models.

Reply for Q3:

Under the greedy protocol, RailCap ties with LNE-blocking but over-suppresses less. On Llama-2 × GSM8K, with one greedy decode per question and SA-PPG on the 0/1 readout (RailCap samples only at capped steps):

StrategySA-PPGUnder-suppressionOver-suppression
No mitigation0.3740.2560.117
LNE-blocking0.3350.0870.248
RailCap0.3320.1140.218

However, a greedy decode, though reproducible, is still a 0/1 readout of a single trajectory: of the 307 questions M_cl solves greedily, 103 have a solve probability below 0.5, and of the 1,012 it fails, 122 have one of at least 0.3. We therefore use the solve probability for all strategies.

(continued in the next comment)

(Part 2/3)

Reply for Q3:

Under the greedy protocol, RailCap ties with LNE-blocking but over-suppresses less. On Llama-2 × GSM8K, with one greedy decode per question and SA-PPG on the 0/1 readout (RailCap samples only at capped steps):

StrategySA-PPGUnder-suppressionOver-suppression
No mitigation0.3740.2560.117
LNE-blocking0.3350.0870.248
RailCap0.3320.1140.218

However, a greedy decode, though reproducible, is still a 0/1 readout of a single trajectory: of the 307 questions M_cl solves greedily, 103 have a solve probability below 0.5, and of the 1,012 it fails, 122 have one of at least 0.3. We therefore use the solve probability for all strategies.

Reply for Q4:

Spread across seeds. We reran all nine settings with three seeds (each seed resamples every model, including M_cl). SA-PPG, mean ± std:

SettingIdentityTEDLNE-blockingRailCap
Llama-2 × GSM8K0.321 ± 0.0040.303 ± 0.0180.291 ± 0.0000.194 ± 0.004
Llama-2 × PQ0.242 ± 0.0020.224 ± 0.0030.302 ± 0.0020.205 ± 0.002
Llama-2 × HumanEval0.416 ± 0.0320.422 ± 0.0440.374 ± 0.0110.255 ± 0.025
Gemma × GSM8K0.309 ± 0.0040.295 ± 0.0090.311 ± 0.0030.197 ± 0.004
Gemma × PQ0.221 ± 0.0020.216 ± 0.0060.294 ± 0.0010.171 ± 0.003
Gemma × HumanEval0.387 ± 0.0300.365 ± 0.0290.344 ± 0.0150.214 ± 0.010
Pythia × GSM8K0.243 ± 0.0030.202 ± 0.0070.282 ± 0.0290.171 ± 0.004
Pythia × PQ0.165 ± 0.0140.169 ± 0.0070.269 ± 0.0100.151 ± 0.002
Pythia × HumanEval0.304 ± 0.0080.276 ± 0.0520.278 ± 0.0200.219 ± 0.003

RailCap has the lowest SA-PPG in all nine settings, and a paired bootstrap over questions shows that its lead over the runner-up is significant in every one (95% CI excludes 0).

Smallest groups and number of groups. With B = 50, the smallest group on Llama-2 × GSM8K has 5 questions. SA-PPG for different B (smallest group size in parentheses):

StrategyB=2 (239)B=5 (80)B=10 (36)B=25 (13)B=50 (5)
Identity0.3200.3200.3250.3270.326
Shortcut0.2230.2400.2450.2470.248
LNE-blocking0.2440.2830.2890.2930.293
RailCap0.1770.1890.1910.1930.191
All-Zero0.4300.4810.4890.4900.490

The ranking is identical for every B, including B = 5, where every group has at least 80 questions, so it does not rely on small groups. And the revised paper adds Figure 3, which computes SA-PPG for every B from 1 to 50 in all nine settings: the curves flatten out well before B = 50, and RailCap is the lowest one across essentially the whole range, so the comparison does not depend on how B is chosen.

Size-dependent weights. Weighting each group by |D_b|^λ interpolates between SA-PPG (λ = 0) and A-PPG (λ = 1):

Strategyλ=0λ=0.25λ=0.5λ=0.75λ=1
Identity0.3260.3370.3500.3650.379
Shortcut0.2480.2490.2480.2450.240
LNE-blocking0.2930.2750.2500.2200.188
RailCap0.1910.1870.1810.1730.163
All-Zero0.4900.4370.3720.2960.219

RailCap is best under every weighting. As λ grows, the frequent low-solve-probability group dominates, and All-Zero, which fails every question, improves steadily until it beats Shortcut at λ = 1. Equal group weights are what prevent a strategy from scoring well by degrading the model.

(continued in the next comment)

(Part 3/3)

Reply for Q5:

Sun et al. (2025). Applied to the mitigated and the clean model, their per-question Hamming distance is A-PPG with a single 0/1 readout, and we will credit the precedent: the case against averaging before differencing is theirs, made for benchmark-rewriting strategies. Their formulation leaves our other two concerns open. It reads each question as a single discrete mark, and since responses sampled for the same question may disagree on correctness, a single discrete mark poorly captures per-question performance (Fig. 2). It also weights every question equally, so the metric is dominated by the solve probabilities that occur most often under the clean model — which is why restoration has to be read at every level of the clean model's solve probability.

MemFree (Ippolito et al., 2023). MemFree bans any token that would complete a 10-gram of the training corpus, to stop verbatim regurgitation. RailCap needs neither the training data nor the leaked text, since its n-gram index is the contaminated model's own greedy trajectory, and it caps the trajectory token at the runner-up instead of banning it, because the goal is to restore M_cl's distribution rather than to block copying. Table 3's "n = 4, ban" is MemFree's hard ban on that index; splitting SA-PPG into the two directions separates the two effects (Llama-2 × GSM8K):

StrategySA-PPGUnder-suppressionOver-suppression
RailCap (cap)0.1910.0940.097
Hard ban (MemFree-style)0.2190.0730.146

Under-suppression is the part of the gap where the mitigated model still solves more than M_cl, over-suppression the part where it solves less. The ban does remove more residual contamination, but it overshoots on questions M_cl can already solve, and the overshoot costs more than the extra suppression gains.

DeconIEP (Chai et al., 2026). DeconIEP trains an embedding-perturbation generator on 400 auxiliary questions to imitate a reference model, backpropagating through the evaluated model, whereas all strategies we compare are training-free. We found no public implementation and reimplemented it from the paper (4-layer generator, ζ = 1e-3, lr = 1e-5, dropout 0.2, |D_aux| = 400 GSM8K training questions, f_ref the pretrained base model). On Llama-2 × GSM8K it leaves contamination essentially intact: leaked accuracy 84.4% → 81.7% against M_cl's 22.9%, and its SA-PPG of 0.312 is the weakest of the strategies we compare (RailCap 0.192, Shortcut 0.237, LNE-blocking 0.291, TED 0.301). We report this as a reimplementation rather than as evidence against the method: the paper does not state the number of training iterations, and its experiments perturb a single answer token on MMLU/TruthfulQA, whereas GSM8K requires the perturbation to redirect a 512-token chain of thought. Its restoration metric, RC = |Acc_mitigated − Acc_clean|, is G-AP and therefore subject to the cancellation in §3.3.

Reply for Q6:

n = 4 was chosen on Llama-2 × GSM8K and then fixed for all other settings without retuning. To check that this choice does not depend on the tuning setting, we repeated the sweep on two settings where n was never tuned (SA-PPG):

SettingIdentityTEDShortcutLNE-blockingn=1n=3n=4n=5n=7
Llama-2 × GSM8K (tuning)0.3260.2950.2480.2930.2430.1990.1910.1970.196
Llama-2 × PQ0.2550.2440.2240.3050.2200.2150.2090.2100.212
Llama-2 × HumanEval0.4330.3910.3390.3370.3420.2780.2700.2640.250

In all three settings, every n ≥ 3 beats all baselines, and n = 4 is within 0.02 of the best n, so RailCap's results do not depend on tuning n.

We thank the reviewer again for these constructive and detailed comments. These additional analyses will be included in the revised version of the paper. We would be grateful for the reviewer's feedback on these clarifications.

(Part 1.1/3) Additional Reply for the Q1: A strategy family with a known ordering. Starting from Identity, we force a failure (solve probability 0) on a share jj of HH, the unleaked questions that M_cl solves with probability at least 0.5. Questions are destroyed in a fixed random order, so a larger jj destroys every question a smaller jj destroys, and nothing else changes.

  • A larger jj is necessarily a worse strategy. A destroyed question's gap becomes rcl(q)r_{cl}(q), which is at least its previous gap ∣rs(q)−rcl(q)∣|r_s(q)-r_{cl}(q)| because rcl(q)≥0.5r_{cl}(q)\ge 0.5. No question moves closer to M_cl, and nearly every destroyed one moves farther. Since these questions are unleaked, destroying them removes no memorization, only genuine capability.
  • Only SA-PPG rises reliably on every model. G-AP and G-APP fall as jj grows on all four models: they reward destroying capability, because the added over-suppression cancels the inflation on leaked questions (OLMo-2-32B: G-AP 0.2153 → 0.0015). A-PPG rises, but on Pythia-12B, whose clean model is weak, by only 0.0018 per row, which is comparable to its sampling noise. Across 1,000 simulated evaluations that redraw 50 samples per question, A-PPG increases strictly over all six rows in 40% of them and SA-PPG in 98%. Both reach 100% on the other three models; G-AP and G-APP reach 0% everywhere.

Rows give jj; j=0j=0 is Identity, and for Llama-2-7B this row matches Identity in Table 1.

Pythia-12B (clean accuracy 0.074, |H| = 25)

Destroyed share of HG-APG-APPA-PPGSA-PPG
0.00.13340.13480.15910.2131
0.20.13100.13280.16090.2413
0.40.12860.13080.16270.2696
0.60.12610.12880.16450.2978
0.80.12370.12680.16630.3255
1.00.12130.12480.16810.3538

Llama-2-7B (clean accuracy 0.219, |H| = 116)

Destroyed share of HG-APG-APPA-PPGSA-PPG
0.00.31920.32430.37930.3261
0.20.30790.31300.38860.3492
0.40.29660.30170.39790.3724
0.60.28480.28990.40750.3966
0.80.27350.27860.41680.4198
1.00.26230.26730.42610.4431

Gemma-4-E2B (clean accuracy 0.227, |H| = 123)

Destroyed share of HG-APG-APPA-PPGSA-PPG
0.00.25400.23220.30940.3147
0.20.24240.22090.31870.3375
0.40.23130.21000.32750.3594
0.60.21980.19880.33660.3820
0.80.20870.18800.34540.4039
1.00.19710.17660.35470.4268

OLMo-2-32B (clean accuracy 0.539, |H| = 362)

Destroyed share of HG-APG-APPA-PPGSA-PPG
0.00.21530.22030.31210.3606
0.20.17290.17790.35030.3868
0.40.12970.13490.38920.4133
0.60.08710.09240.42760.4395
0.80.04410.04950.46630.4661
1.00.00150.00700.50470.4924

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.