- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how to evaluate and improve contamination mitigation, where a decoding-time intervention is supposed to bring a contaminated model’s benchmark performance back to that of a clean reference model. On the metric side, it argues that the prevailing G-AP metric has three flaws. It relies on a noisy single-sample 0/1 readout, it averages before differencing so that over- and under-suppression cancel, and it weights questions uniformly, which lets a trivial All-Zero strategy look competitive when the clean model is weak. The authors build up G-APP, A-PPG, and finally SA-PPG, which uses sampled solve probabilities, takes absolute per-question gaps against the clean model, and averages these gaps equally across bins of the clean model’s solve probability. On the method side, the paper proposes RailCap. RailCap indexes the contaminated model’s greedy trajectory by n-grams and, whenever a sample re-enters that trajectory, caps the logit of the next trajectory token at the runner-up value. Experiments use LoRA-simulated contamination on Llama-2-7B, Gemma-4-E2B, and Pythia-12B across GSM8K, a paraphrased GSM8K variant (PQ), and HumanEval. Under SA-PPG, RailCap is best in all nine settings, and the ranking of prior methods changes substantially relative to G-AP, most notably for LNE-blocking.
Critical strengths
- A precise and useful diagnosis of the prevailing metric. The sign decomposition shows cleanly that G-APP ≤ A-PPG and that a zero aggregate gap can come entirely from cancellation. Table 1 makes this concrete, since LNE-blocking’s near-zero G-AP hides an over-suppression component twice as large as RailCap’s. Figure 2 adds solid evidence that single-sample 0/1 readouts are not reproducible even for the same model. Several published mitigation papers report G-AP as their headline number.
- A simple mitigation with consistent results. RailCap needs no auxiliary detector, no neuron localization, and no access to the clean model. It costs one greedy decode and an n-gram lookup per question. It outperforms all baselines across three models and three domains, and the observation that the entropy-based LNE-blocking falls below Identity on PQ is a valuable finding in its own right.
Critical weaknesses
-
SA-PPG has a loophole of the same kind it was designed to close, and its validation is circular. Equal weighting across bins implicitly assumes a uniform prior over clean solve probabilities, so a constant strategy that sets every solve probability to 0.5 scores about 0.25. On Llama-2 × GSM8K this beats Identity, TED, and LNE-blocking while ignoring the questions entirely. SA-PPG thus replaces the All-Zero loophole with a collapse-to-the-middle one. Its support also rests largely on RailCap winning under it, which is circular without an independent test against known ground truth.
-
RailCap’s claim of needing no contamination estimate is overstated. RailCap treats the greedy trajectory as the contamination signal and applies the cap to every question. On unleaked questions where the model is confidently correct, samples also follow the greedy path, so RailCap likely suppresses genuine capability there, which is the failure mode the paper attributes to estimate-based methods. Without Δ⁺ and Δ⁻ split by leaked and unleaked questions, it remains unclear whether RailCap removes memorization or simply injects diversity
Questions for authors
- What SA-PPG does a constant strategy r ≡ c attain for c in [0, 1] in each setting? If c = 0.5 outperforms most prior methods, how do you reconcile this with the argument that SA-PPG fixes the All-Zero loophole of A-PPG?
- Can you report Δ⁺ and Δ⁻ separately for leaked and unleaked questions for RailCap and each baseline? In particular, how much does RailCap reduce solve probability on unleaked questions relative to Identity?
- The absolute per-question gap is biased upward under finite sampling, and Figure 2 already shows a clean-versus-clean gap of about 0.041 at m = 50. What is the SA-PPG of the clean model against itself in each of the nine settings, and are RailCap’s margins over the runner-up larger than this floor and than variation across seeds?
- Equal bin weighting gives a bin with one or two questions the same weight as a bin with hundreds. How many questions fall in each bin, and how sensitive are the rankings to coarser or equal-frequency binning? Are the bins defined using the same clean-model samples that enter the gaps?
We thank the reviewer for the careful review. We address the concerns below.
Reply for W1:
Our validation is not circular.
Our contribution is not incremental.
Reply for W2:
The runner-up pattern (Fig. 1b) holds across models, contamination types, and benchmarks. The table gives the share of forks at which the clean model's token is the contaminated model's runner-up, at the k-th fork of each leaked question. From the third fork on, this share is 52–63% on GSM8K and PQ and above half on HumanEval.
Each component of RailCap reduces one error component. We justify each design choice from an evolutionary perspective: starting from LNE-blocking, we swap in RailCap's components one at a time and track which error component each reduces (Llama-2 × GSM8K):