- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper argues that removing exact duplicates is insufficient for benchmark decontamination, and proposes a graph based decontamination policy that covers a wider range of decontamination edge types (exact, paraphrase, answer leakage, template, benign). They perform intervention experiments by retraining with different pruning policies, and they measure how the NLL gap closes against baselines. Authors also perform human review to verify the precision on the natural pool.
Critical strengths
The experiment control design seems to be thorough. The authors also report negative results such as 32B gate. However, the experimental design's strength is contingent on proper description of implementation details that are not sufficiently provided in the paper (see weakness section).
Critical weaknesses
The premise of this paper relies on the understanding that exact duplicate removal is insufficient for decontamination, this is reasonable but already widely established in the field (Yang et al, 2023; Jiang et al, 2024). This paper also does not cite highly relevant works with overlapping research problems (Yang et al, 2023; Jiang et al, 2024). For instance, Jiang et al, 2024 used counterfactual training to measure contamination effects, which overlaps the framework in this paper. Authors need to cite these papers and establish the novelty of their contribution over prior work.
Many terms are named but not defined. Examples of missing details include edge-type features (how is it computed?), thresholds (numbers?), and severity rules. A major example of this is the cutoff value for graph-pruning, I do not see this cutoff declared in the paper. Also, some important methods are not explained, just named. For example, what exactly is “graph-structured” contamination in table 7? I can only find one mention of this word in the text (line 143), with no further explanation. Another similar case is “retrieval-leak”, there are many cases like these that the authors need to fix to make the paper clear and reproducible.
Even though the authors disclose the use of LLMs for writing assistance, the style of writing in this paper sounds highly “AI-like”, making it overly verbose and difficult to read. The overly complex sentence structure and overuse of noun-stacking are signs of AI-writing that make the paper difficult to read. Throughout the paper, the authors overuse semi-colons in their writing. The worst offending paragraph is the last paragraph in the introduction section (line 54-60). This paragraph is one long sentence that presents a list of 4 items separated by semi-colons, and each individual item is itself a comma separated list. Even though it is grammatically correct, this kind of long, list-of-lists sentence is difficult to understand for a human reader. The authors should take time to revise the writing style in this paper to communicate their ideas in a more clear and straightforward fashion. As it stands, it reads like an AI-generated paper with insufficient human auditing.
Citations: Yang, Shuo, et al. "Rethinking benchmark and contamination for language models with rephrased samples." arXiv preprint arXiv:2311.04850 (2023). Jiang, Minhao, et al. "Investigating data contamination for pre-training language models." arXiv preprint arXiv:2401.06059 (2024)
Questions for authors
Table 1 says Typed Graph recovers 100% of controlled edges that were injected. So why does the neural graph arm consistently underperform the oracle (not seed noise because it’s consistent across seeds in supp.table 4), can you report edge recall and precision for neural-tier contamination? It is also unclear to me where I can see the deletion count for the neural arm?