- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
An audit idea for hand-written symbolic ARC solvers: freeze the whole pipeline at a commit, refit its learned parts (here a Naive-Bayes proposer) on an 800-task dev split of ARC-AGI-2 training, then compare task-exact success on 200 reserved training tasks (pH) with the 120 public evaluation tasks (pE). Eval is harder, so pE > pH is suspicious. Decision rule: flagged / consistent / inconclusive from a Newcombe interval on the difference, δ=0. On the authors' own solver LGE, the frozen πnarrow solves 31.7% of eval vs 5–9% of reserved training (gap +18.7 to +26.7 points over five splits), flagged. 44 of the 54 selected executors were committed in April 2026 during eval iteration. They also recount their older provenance check and find two errors: 1,800 support records are only 1,033 distinct task IDs, and 6 eval tasks sit in the ARC-AGI-1 eval "support" pool, so eval tasks supported themselves. With manifests selected without eval outcomes, nested selection and an independent solver (ARGA, patched), eval is 0/120 and the gap reverses; all "inconclusive". Power: an 80% flag probability needs a 9–11 point excess. Appendices add a Hoeffding bound (not certified at their pool sizes), a checklist, LGE on other benchmarks (BBH 84%, SCAN 100%) and frontier-model counts on the same 120 tasks (e.g. GPT-5.4 pro 96/120).
Critical strengths
- Real self-audit: errors admitted and fixed (duplicate IDs, eval tasks counted as their own support, a content-level duplicate db695cfb/55783887). The lesson to enforce independence by task identity and content is concrete.
- Statistical honesty: exact power computation, "inconclusive is not a pass", no fake "consistent", split sensitivity. Rare.
Critical weaknesses
- The audit is not validated. One positive system, contaminated by the authors' own account. Everything else is "inconclusive" at 0/120, which carries no information, and "consistent" never happens. No controlled experiment (inject eval-derived executors in known amount, run a clean solver) so no false positive/negative rate. At 120 eval tasks it only catches a ≥9 point excess. The paper therefore shows that "our old pipeline did suspiciously well on eval", which the authors knew before they started. Calling this a contamination audit method is oversold.
- Irrelevant to where ARC is happening. LGE solves nothing on eval after correction and is far behind (Appendix H: frontier models solve 40–80% of the same tasks). The audit does not apply to LLM or test-time-training solvers (Table 2 rows are "not results"), because refit can't remove pretraining exposure. The core test, frozen system on held-out train vs harder set plus a difficulty fudge, is a basic generalization-gap check.
Questions for authors
- Build controlled contaminated and clean variants and report flag rates.
- Hand-written code saw all of training. How do you know pE>pH isn't just that you studied eval tasks more closely, which the refit doesn't address?
- Why δ=0 when your difficulty model predicts ~1% on eval?