- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ARC-∞, a framework for automatically generating sequences of ARC-style reasoning worlds to evaluate conceptual bootstrapping. An LLM-based Designer recursively extends earlier environments, with hyperparameters controlling the depth and breadth of conceptual growth. The authors evaluate coding-agent harnesses and find that persistent memory improves learning, while an additional library-learning step that consolidates accumulated code into reusable functions can further improve performance.
Critical strengths
The paper explores an interesting direction: evaluating intelligence through cumulative concept acquisition rather than independent reasoning problems. The automated benchmark-generation framework is promising, and the separation of conceptual planning from implementation is a thoughtful design choice. The analyses of how earlier computations become components of later tasks, and why agents fail to reuse them, are insightful.
Critical weaknesses
-
Benchmark correctness / ceiling performance: Appendix C.3.2 identifies a significant flaw in the main evaluation worldline: at age 3, target outputs can depend on hidden random values that cannot be inferred from the input, affecting 17/20 problems. Other generated worlds repeatedly introduce essentially the same concepts. These issues raise concerns about benchmark validity and suggest the need for stronger automatic verification or a baseline for the ceiling performance the benchmark can support.
-
Harness selection and the interpretation of library learning: The proposed library learning is essentially an additional LLM session that refactors accumulated code into reusable functions and checks earlier examples. It is unclear how much this differs from standard coding-agent memory management and refactoring. Meanwhile, recent evidence from OpenAI (2026) shows that reasoning retention and context compaction can substantially improve ARC-AGI-3 performance. This raises the possibility that compaction already performs some form of implicit knowledge consolidation. The authors should clarify whether their harnesses use these capabilities and compare against stronger coding-agent baselines with compaction and periodic refactoring / reasoning trace visibility in the next turn.
-
Why should the learned library consist of code rather than natural-language concepts? An interesting aspect of ARC-∞ is that its own Designer uses natural language to represent previously acquired concepts and propose new conceptual compositions. The Director maintains descriptions of earlier concepts and generates natural-language growth intents, which the Creator subsequently translates into executable operations. This suggests that natural language may itself be an effective representation for accumulated conceptual knowledge, potentially supporting more flexible abstraction and transfer than executable functions alone. Yet the learner-side library learning focuses exclusively on consolidating Python code. It would be valuable to compare libraries containing (1) natural-language descriptions of learned concepts and rules, (2) executable functions, and (3) both. Such a comparison would help clarify whether conceptual bootstrapping specifically benefits from executable program composition, or whether maintaining and reusing explicit conceptual representations is sufficient.
Ref: OpenAI (2026). How enabling two settings tripled our scores on the ARC-AGI-3 benchmark. https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores/
Questions for authors
-
Why choose code as the primary representation? Could natural-language concept descriptions, or a hybrid code–language library, support comparable or better bootstrapping?
-
Could compaction itself provide implicit knowledge consolidation, and how would the proposed library learning compare against a standard coding agent with compaction and periodic refactoring? e.g., the arc-code is not a good harness in this sense.
6 comments
W1-3 & W1-4. These issues suggest the need for stronger automatic verification or a baseline for the ceiling performance the benchmark can support.
We provide both. A solver given the true operations sets a baseline for the ceiling, and we added automatic checks over every test problem. To estimate the ceiling, we provided a solver with the world's true operations, its training pairs and the problem's three support pairs, but not the program or the answer. In this experiment, the solver is correct on 219/220 (main) and 211/220 (second) problems. This is what a current LLM reaches when it is given the true operations. A learner that acquires these operations from the training pairs could therefore reach at least this score with a current LLM. A more capable LLM or harness could compose the operations better, so we expect the ceiling to be almost perfect. The best harness reaches 150/220 on the main worldline, so the benchmark leaves clear room above current harnesses.
We also added three automatic checks over every test problem of both evaluation worldlines, and we will build them into the generator so that every new world is checked before it is accepted. For hidden values, changing the generator's random values with the input fixed changes no answer after the fix. For ambiguity, alternatives that fit all three supports but change the answer exist in only 8/220 (main) and 22/220 (second) problems (ages 0–10). For repetition, the previous world's operations reproduce 97 of the 99 examples of the repeated worlds in Appendix C.3.1, so such a world can be rejected automatically.
W2-1. How does library learning differ from standard memory management and refactoring?
Library learning differs from refactoring in its goal. We ran the comparison the reviewer suggested, and library learning scored higher in every setting.
Both run a second session over the kept code, so they look alike. They differ in their goal. Library learning is an established method in program learning (DreamCoder, Ellis et al., 2021; LILO, Grand et al., 2024). It builds functions from solved problems so that later problems can be solved by composing them. Refactoring instead changes the internal structure of code without changing its observable behavior (Fowler, 2018). Our step follows the goal of library learning. It replaces values that the learner memorized for each example with a computation from the input. For example, at age 5 the learner first stored the masks of each shape it had seen. Library learning replaced this table with a function that folds the edge cells of any shape into a mask (Appendix E.4). Refactoring has no reason to remove such a table, because the table already reproduces every example.
If the two were the same, they would score the same on ARC-∞. We added a second session after each world. It either refactors the code and notes or applies our library-learning instruction. Refactoring organizes the code, removes duplication, extracts helpers and runs regression checks. Both use the same evidence, budget and test interface.
| Main worldline, correct at ages 0–5 (/120) | Refactoring | Library learning |
|---|---|---|
| Code and notes, new conversation for each world | 49 | 54 |
| Code and notes, one long conversation with compaction | 50 | 88 |
| Library learning keeps code only (the best format, see W3) | 49 | 85 |
In other words, harnesses that keep memorized examples overfit to each world (Finding 2), and library learning helps because it turns these examples into rules. We do not claim that library learning is the best learner. ARC-∞ is built to tell such strategies apart, and this comparison shows that it does.
W2-2. Do the harnesses use reasoning retention and compaction?
Yes, the harnesses use both reasoning retention and compaction. All harnesses run on Codex, which keeps the model's reasoning across tool calls within a session. We set no truncation. Schema keeps one conversation across all worlds and compacted it 60 times.
W2 & Q2-3. Could compaction already consolidate knowledge? How does library learning compare with compaction and periodic refactoring?
Compaction alone does not consolidate knowledge, and on top of compaction, library learning helps far more than refactoring (88 vs. 50). If compaction already consolidated knowledge, Schema would not overfit. It still overfits each world (Sec. 4.2) and scores 71/220, against 91 for Basic. In the table above, the long conversation with compaction reaches 88/120 with library learning and 50/120 with periodic refactoring. Compaction shortens the conversation, and library learning changes the knowledge that the learner keeps, so the two work together.
W2-4. Comparison with stronger coding-agent baselines
Schema and Tycho are our stronger coding-agent baselines. Both report near-perfect results on the public ARC-AGI-3 games, and both are among the harnesses we evaluate (Sec. 4.2). If the reviewer suggests another harness, we would be glad to run it on ARC-∞. As we noted above, finding the strongest harness is left to those who use the benchmark.
W3 & Q1. Why code rather than natural language? Could natural-language or hybrid libraries do as well?
We tried all three, and code worked best. We ran the three libraries the reviewer suggested with the same library-learning step and budget, so only what is kept between worlds differs. The natural-language learner can still write and run code within a world and at test time.
| Kept between worlds | Correct, ages 0–5 (/120) |
|---|---|
| Code | 85 |
| Natural-language descriptions | 74 |
| Both | 54 |
Code can be run, so it can be checked against examples and combined directly, while a description must be turned into code again in every new world. Recent agent work finds the same. Executable code serves as a better skill library and action format than text (Voyager, Wang et al., 2023; CodeAct, Wang et al., 2024). On ARC, writing programs is better than predicting answers directly at precise computation and at composing several concepts (Li et al., 2025). The Designer works the same way. The Director plans in natural language, and the Creator turns each plan into code.
We hope these results address the reviewer's concerns. We welcome any further questions or requests for additional experiments.
References
- Ellis, K., et al. (2021). DreamCoder: Bootstrapping Inductive Program Synthesis with Wake-Sleep Library Learning. PLDI 2021. link
- Fowler, M. (2018). Refactoring: Improving the Design of Existing Code (2nd ed.). Addison-Wesley. link
- Grand, G., et al. (2024). LILO: Learning Interpretable Libraries by Compressing and Documenting Code. ICLR 2024. link
- Jimenez, C. E., et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024. link
- Li, W.-D., et al. (2025). Combining Induction and Transduction for Abstract Reasoning. ICLR 2025. link
- Wang, G., et al. (2023). Voyager: An Open-Ended Embodied Agent with Large Language Models. link
- Wang, X., et al. (2024). Executable Code Actions Elicit Better LLM Agents (CodeAct). ICML 2024. link
- Yang, J., et al. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS 2024. link
We thank the reviewer for the detailed review. We ran both suggested comparisons. One compares library learning with refactoring and compaction, and the other compares code with natural-language memory. We also fixed the age-3 world and reran every harness. The fixed world, the new checks and the comparison with refactoring are in the revised PDF in red. Below, the main worldline is the one used for the learners in Sec. 4.2, and the second worldline is the one in Appendix F.
Before addressing each point, we would like to clarify the scope of the paper. Like other benchmark papers, it proposes a benchmark, and how to solve the benchmark is not its main focus. Our harnesses are reasonable baselines that show that the benchmark works as intended, and designing a state-of-the-art learner is not our goal. For example, the ARC-AGI paper (Chollet, 2019) proposes the benchmark without proposing the best solver. SWE-bench (Jimenez et al., 2024) likewise reported simple baselines, and stronger agents such as SWE-agent (Yang et al., 2024) came in later work. Building better learners is left to those who use the benchmark. Within this scope, we ran every comparison the reviewer requested.
W1-1. At age 3 of the main worldline, answers depend on a hidden value (17/20 problems)
This is a single case that we found and reported ourselves in Appendix C.3.2. We fixed it, reran every harness, and all three findings hold. The answers at age 3 depended on a socket color that the input never showed. The input now shows the color through an opening in the cover, which also fixes the two age-4 problems that reuse this rule. We reran all five harnesses from age 0 and report both results.
BasicBasic w/o memorySchemaTychoBasic w/ libraryMemory still helps (91 vs. 76), library learning is the most accurate and the cheapest (2.36M vs. 3.62M output tokens for
Basic), and all harnesses still fail in the last worlds (at most 7/20 at ages 9–10). This case occurred only at age 3 of the main worldline. The generator's instructions already forbid hidden values, and its review tests for them, but the review missed this case at age 3 (Appendix C.3.2). Run automatically over every test problem of both evaluation worldlines, the same test finds no other hidden value, and we will make it an automatic step of the generator.W1-2. Other generated worlds repeatedly introduce essentially the same concepts.
This refers to the worldline we reported as a failure case in Appendix C.3.1, and such repetition is rare. That worldline was not used for evaluation and does not affect any reported result. If generated worlds generally repeated earlier concepts, many worlds would add nothing new. In the 10 worldlines we grew to age 50, only 2 of the 500 worlds (0.4%) add no new function or only renamed copies of earlier ones. In the evaluation worldlines, 74% (main) and 70% (second) of the problems need an operation introduced in their own world. Therefore, the repetition in Appendix C.3.1 is a rare failure, not a general tendency.