acceptodds
Under review as a conference paper at ICLR 2027

Freeze, Refit, Compare: A Procedure-Level Contamination Audit for Symbolic ARC Solvers

Abstract

Symbolic solvers for the Abstraction and Reasoning Corpus (ARC) build their answers from libraries of hand-written operations, so every solution can be read. Readable code can still hide evaluation contamination: iterating on the public evaluation tasks can leave traces of them in operation bodies, validators and special cases, invisible to a check of method names for evaluation identifiers. We propose an audit of the whole executable procedure: freeze the solver, refit its trainable part on a development split of the training set, and compare success on reserved training tasks with success on the public evaluation set. Because the evaluation set is deliberately harder, an evaluation excess is a warning sign. On our own solver, the frozen pipeline's evaluation rate was 3.3 to 6.3 times its reserved-training rate on each of four splits, and commit history points to operations written during evaluation iteration as the likely source. A recount of our own provenance check found two counting errors, including evaluation tasks counted as their own held-out support because they reappear in ARC-AGI-1. With support recounted, manifests selected without evaluation outcomes and nested controls reverse the gap, as does an independently written solver; under our rule these outcomes are inconclusive, not clean passes, and for the frozen manifests the reversal is consistent with the measured difficulty shift. Under an independent-task model at reserved-training rates of 5 to 9%, the audit flags an evaluation excess with 80% probability only when the excess reaches about 9 to 11 percentage points. Inspectable is not the same as clean: contamination can hide in code that nobody calls trained, and an audit costing one training split can expose it.

Primary area:
datasets and benchmarks
Keywords:
ARC-AGI-2, benchmark contamination, evaluation audit, data leakage, program synthesis, symbolic reasoning, adaptive data analysis, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_AsTaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

An audit idea for hand-written symbolic ARC solvers: freeze the whole pipeline at a commit, refit its learned parts (here a Naive-Bayes proposer) on an 800-task dev split of ARC-AGI-2 training, then compare task-exact success on 200 reserved training tasks (pH) with the 120 public evaluation tasks (pE). Eval is harder, so pE > pH is suspicious. Decision rule: flagged / consistent / inconclusive from a Newcombe interval on the difference, δ=0. On the authors' own solver LGE, the frozen πnarrow solves 31.7% of eval vs 5–9% of reserved training (gap +18.7 to +26.7 points over five splits), flagged. 44 of the 54 selected executors were committed in April 2026 during eval iteration. They also recount their older provenance check and find two errors: 1,800 support records are only 1,033 distinct task IDs, and 6 eval tasks sit in the ARC-AGI-1 eval "support" pool, so eval tasks supported themselves. With manifests selected without eval outcomes, nested selection and an independent solver (ARGA, patched), eval is 0/120 and the gap reverses; all "inconclusive". Power: an 80% flag probability needs a 9–11 point excess. Appendices add a Hoeffding bound (not certified at their pool sizes), a checklist, LGE on other benchmarks (BBH 84%, SCAN 100%) and frontier-model counts on the same 120 tasks (e.g. GPT-5.4 pro 96/120).

Critical strengths
  • Real self-audit: errors admitted and fixed (duplicate IDs, eval tasks counted as their own support, a content-level duplicate db695cfb/55783887). The lesson to enforce independence by task identity and content is concrete.
  • Statistical honesty: exact power computation, "inconclusive is not a pass", no fake "consistent", split sensitivity. Rare.
Critical weaknesses
  • The audit is not validated. One positive system, contaminated by the authors' own account. Everything else is "inconclusive" at 0/120, which carries no information, and "consistent" never happens. No controlled experiment (inject eval-derived executors in known amount, run a clean solver) so no false positive/negative rate. At 120 eval tasks it only catches a ≥9 point excess. The paper therefore shows that "our old pipeline did suspiciously well on eval", which the authors knew before they started. Calling this a contamination audit method is oversold.
  • Irrelevant to where ARC is happening. LGE solves nothing on eval after correction and is far behind (Appendix H: frontier models solve 40–80% of the same tasks). The audit does not apply to LLM or test-time-training solvers (Table 2 rows are "not results"), because refit can't remove pretraining exposure. The core test, frozen system on held-out train vs harder set plus a difficulty fudge, is a basic generalization-gap check.
Questions for authors
  1. Build controlled contaminated and clean variants and report flag rates.
  2. Hand-written code saw all of training. How do you know pE>pH isn't just that you studied eval tasks more closely, which the refit doesn't address?
  3. Why δ=0 when your difficulty model predicts ~1% on eval?
Reviewer_3hitOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper examines whether there is test-set contamination in a bespoke ARC-AGI-2 solver called LGE. The authors propose a method that involves refitting the learned proposer and ranker on the training set, freezing the remaining components, and measuring the performance gap on held-out training tasks and public evaluation tasks. They find that LGE performs substantially better on the public evaluation set despite its greater difficulty, suggesting potential contamination in the frozen, hand-written components.

Critical strengths
  • The core spirit of the paper is an important direction for research: how can we detect when new published models and results have overfit to the test data?
Critical weaknesses
  • The presentation in this paper is poor, it is difficult to read and understand the paper’s setup and key results.
  • The demonstrated breadth of the paper’s analysis is extremely narrow; the paper only examines a single solver, LGE, which is the authors’ own system and is neither state of the art nor widely used. The paper also does not demonstrate that LGE is similar enough to state-of-the-art solvers in more practically useful tasks for the method to be easily transferred.
  • The ARC task is primarily used now to measure the abstract reasoning capabilities of general-use LLMs; yet Table 2 shows that the paper’s method cannot be applied to the zero-shot language model setting. It is unclear why anyone should care about the performance of symbolic solvers on ARC.
Questions for authors
  1. What insights can this method give into zero-shot language model performance?

  2. Why should people care about the performance of symbolic solvers on ARC, which is mostly used to measure the abstract reasoning capability of general-purpose LLMs?

  3. Are there specific tasks/domains where symbolic solvers are still widely used, where the paper’s method would apply to generate useful insights?

  4. What does “LGE” stand for? Please make the explanation for this pipeline clearer, and consider making Figure 3 of the LGE pipeline more prominent.

Reviewer_DGQoOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes "Freeze, Refit, Compare", a method for detecting evaluation contamination in symbolic ARC solvers, where hand-written operations may contain information from public evaluation tasks. The method freezes the solver, retrains its trainable components on part of the training set, and compares performance on held-out training tasks and harder public evaluation tasks. The authors test the method on their own solver, LGE, which performs 3.3 to 6.3 times better on evaluation tasks than on held-out training tasks. The paper also finds errors in the earlier contamination checks, such as counting duplicate tasks as independent examples. When operations are selected without using eval results, the solver performs worse on eval tasks than on held-out training tasks, and similar results are observed for ARGA solver. The paper states the method could identify possible contamination, but can't prove that the detected differences are caused by contamination.

Critical strengths
  • The paper addresses evaluation contamination in hand-written solver code, which is an important problem. The paper proposes "freeze, refit, compare" procedure that freezes the solver, retrains its learned components, and compares performance on held-out training and public evaluation tasks to detect potential contamination, which is a promising approach to solve the problem.
Critical weaknesses
  • The presentation of the paper is poor. It is very difficult to read, even the abstract. Including "What this paper is not" paragraph in the introduction section does not add much value. Section 5.2 is particularly difficult to follow. The paper needs substantial rewriting to improve clarity. Additionally, it lacks clear visual plots that would help understand different ablations and results. Section 4, "The Audit", is poorly explained, without sufficient motivation for the method. Overall, much of the manuscript reads as a technical description of the procedures, and the provided figures do little to clarify the main findings of the paper.

  • Limited methodological novelty and limited evaluation. The proposed method relies on standard held-out evaluation and statistical tests, and it is unclear what is novel other than applying it to symbolic solvers. The method is mainly tested on the internal solver, and ARGA experiment provides limited additional evidence. Although the paper includes some ablations, there are no controlled experiments with known contaminated and uncontaminated solvers to measure detection accuracy and false positive rates. It is unclear how to distinguish contamination from differences in task difficulty and solver capabilities.

Questions for authors
  • Q1. What is the main novelty compared with existing contamination detection methods and standard held-out validation?
  • Q2. Could you provide more experiments to evaluate the method on controlled contaminated and uncontaminated solvers to measure the reliability of it?
  • Q3. How do you distinguish contamination vs differences in task difficulty or vs solver capabilities?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.