- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces ARC-Golf Bench, a 96-task benchmark for continual program optimization. It is built from the NeurIPS 2025 Google Code Golf Championship (ARC-AGI-1 grid transformations, scored by Python program length in bytes). Tasks are selected by fitting a graded response model (GRM) to the final submissions of 48 human teams. Each task’s bytes are expressed as a ratio to the best human solution, binned into five ordinal levels, and used to estimate discrimination and difficulty.
The paper also proposes Golf-UnZip, an LLM-judge procedure that expands golfed programs one verified rewrite at a time. From it, the authors derive Trick Richness, Method Diversity (rubric-based signatures), and a verification-before-submit rate, and they report model-specific harness effects on these with bootstrap and BH-corrected tests.
The paper further studies cross-task experience transfer by distilling optimization skills from 50 source tasks. On ten held-out tasks, the library reduces total bytes by 13.2% for one worker, while combining it with a lane-conditioned harness yields a 9.4% improvement for another worker over a matched library-free parallel control.
Critical strengths
-
A clean optimization objective and carefully engineered evaluation protocol. The byte-length objective is deterministic and avoids runtime measurement noise. The evaluation harness includes isolated, version-pinned containers, controlled information access, independent verification, saved-candidate recovery, and retained execution traces. The eight model–harness configurations are evaluated across 96 tasks with three runs per configuration, and failures are explicitly accounted for.
-
A useful model–harness comparison with behavioral analysis. The crossed evaluation reveals substantial model-dependent harness effects, including reversals in model rankings. The trajectory and behavioral analyses offer a more informative picture of sustained optimization than terminal leaderboard scores alone.
Critical weaknesses
- Positioning relative to the closest prior work is missing, which weakens the novelty claims. The framing (moving beyond pass/fail toward sustained, verifier-guided optimization, anchored to human competition results, with trajectory tracking and scaffold comparisons) closely follows ALE-Bench[1], which is not cited. The overlap goes beyond motivation:
- ALE-Bench also places agents on a human leaderboard scale, tracks score trajectories over a run, and provides a curated subset whose agreement with the full set it validates.
- The intervention in Sec. 6.3 (parallel sessions assigned to distinct solution families with family-specific guidance, keeping the best verified candidate) resembles ALE-Agent’s diversity-oriented search with targeted improvement prompts, which ALE-Bench already identified as its main source of gains.
- The human-only interpretation of the competition reference is not justified, which weakens the human-relative performance claims. The reference set is not purely human. Public competition write-ups indicate that several top teams used LLMs substantially; one top-5 team’s submission was reportedly almost entirely LLM-generated, and others started from LLM-generated solutions. Yet “human teams”, “human-equivalent rank”, and “improves on the best human reference on 10 tasks” are central claims.
[1] Imajuku, Yuki, et al. "Ale-bench: A benchmark for long-horizon objective-driven algorithm engineering." Advances in Neural Information Processing Systems 2025.
Questions for authors
-
How do you position the benchmark and the lane-conditioned harness relative to ALE-Bench’s long-horizon iterative-refinement setting and ALE-Agent’s diversity-oriented search?
-
In line 239, you mentioned the gaps between the best and worst configurations are 4.2 on easy tasks and 8.6 on hard tasks. However, in table 1, easy: 13.0-9.1=3.9, hard: 20.6-11.3=9.3. Which one is correct?
-
In line 240 and line 241, you mentioned "Qwen, DeepSeek, and GLM receive worse ranks as difficulty increases under both harnesses". However, in table 1, Codex·DeepSeek-V4-Flash configuration, the rank is: 15.5 (medium) -> 14.8 (hard).
-
About the validity of the IRT subset. The tasks are selected for discriminating among human teams. Do they also discriminate among agents? Have you checked whether configuration rankings on the 96-task subset agree with rankings on a larger set of tasks?
-
How do you position ARC-Golf Bench relative to ALE-Bench's long-horizon iterative-refinement setting, human-relative evaluation, and ALE-Agent's diversity-oriented search? What are the specific methodological contributions beyond changing the optimization objective to program length?
2 comments
Thank you for the clarifications. I still have two concerns:
-
Does the 96-task subset work well for evaluating agents? The authors found no significant correlation between human IRT discrimination and agent discrimination (ρ = −0.13, p = 0.21). This makes me question whether IRT is a good way to select tasks for evaluating agents. I also checked the response to Reviewer Hmdz. The 96- vs. 400-task comparison only covers competition teams, not agents. The agent results compare different scoring methods on the same 96 tasks. So we still don't know whether agent rankings would hold on a larger set of tasks, or whether this subset is better than 96 randomly selected tasks. It would also help to compare the differences between configurations with the run-to-run variation within each configuration.
-
I appreciate the authors' willingness to change "human" to "competition teams." The competition results are still valid reference points. However, some teams used LLMs, and their time and compute budgets were different from those of the evaluated agents. This means the results cannot be presented as a direct comparison between agents and humans.
Thanks for reading our paper.
Response to Q1 (and Q5, W1).
Thank you for pointing out ALE-Bench; we will cite and discuss it in the paper. Both benchmarks evaluate long-horizon iterative optimization using fine-grained, human-referenced scores. However, they differ in several aspects:
Thus, our work complements ALE-Bench by focusing on the mechanisms of iterative optimization and the transferability of optimization skills, rather than proposing another search algorithm.
Response to Q2 and Q3.
Thank you for catching these inconsistencies.
The values in Table 1 are correct: the gaps should be 3.9 (easy) and 9.3 (hard). The original values (4.2 and 8.6) came from an write-up of earlier results. We inadvertently left the outdated numbers in the text and will correct them. We have also carefully rechecked the manuscript for similar inconsistencies.
We agree that our statement was imprecise. We intended to describe the overall trend rather than a strictly monotonic increase. For example, Codex·DeepSeek-V4-Flash worsens from 9.4 (easy) to approximately 15 (15.5 medium, 14.8 hard), despite a slight improvement from medium to hard. We will revise the statement to clarify that these models generally perform worse on medium and hard tasks than on easy tasks, without implying monotonic degradation.
Response to Q4
(a) Agent discrimination. Yes, the selected tasks also discriminate among agents. Across the 96 tasks, the average best–worst rank gap among eight configurations is 10.83 (easy), 11.69 (medium), and 13.36 (hard) rank positions. However, human-calibrated IRT discrimination does not significantly correlate with agent discrimination (Spearman ρ = −0.13, p = 0.21). Thus, while the subset effectively distinguishes agents, we do not claim that human discrimination directly predicts agent discrimination.
(b) Ranking robustness. Check the response to reviewer 2 (reviewer Hmdz) on Q2.
Response to W2
We agree that some competition teams used LLMs, so describing their submissions as purely human-generated is imprecise. Although these results likely involved human guidance and optimization, we cannot verify the degree of human contribution. We will therefore use “competition teams,” “competition-relative rank,” and “best competition reference” throughout the paper, and explicitly acknowledge potential LLM assistance.
Importantly, the validity of competition-relative comparisons does not depend on the submissions being purely human-generated, as they remain independently verified performance references.