acceptodds
Under review as a conference paper at ICLR 2027

ARC-: Automating the Design of Recursively Growing Intelligence Tests

Abstract

Conceptual bootstrapping, using known concepts to acquire new ones, is a hallmark of human intelligence, yet existing intelligence tests cannot measure it: their tasks are independent or fixed, and once solved, leave nothing to measure. Measuring bootstrapping requires a test that grows recursively, with each environment building on the concepts of earlier ones, which is impractical to design by hand. We introduce the Abstraction and Reasoning Continuum (ARC-), a framework that automates the design of recursively growing intelligence tests. From a seed world, a Designer agent repeatedly produces the next world, yielding an unbounded worldline whose depth and breadth are controlled by two hyperparameters. We show that these controls steer growth as intended and that concepts accumulate across worlds. Evaluating state-of-the-art agent harnesses along a generated worldline, we find that they bootstrap from earlier worlds but fail to sustain it, and that a simple library learning step that consolidates prior knowledge extends their bootstrapping.

Primary area:
datasets and benchmarks
Keywords:
Intelligence Evaluation, Automated Benchmark Generation, Conceptual Bootstrapping, Open-Ended Generation, World Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_A8XXOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ARC-∞, a framework for automatically generating sequences of ARC-style reasoning worlds to evaluate conceptual bootstrapping. An LLM-based Designer recursively extends earlier environments, with hyperparameters controlling the depth and breadth of conceptual growth. The authors evaluate coding-agent harnesses and find that persistent memory improves learning, while an additional library-learning step that consolidates accumulated code into reusable functions can further improve performance.

Critical strengths

The paper explores an interesting direction: evaluating intelligence through cumulative concept acquisition rather than independent reasoning problems. The automated benchmark-generation framework is promising, and the separation of conceptual planning from implementation is a thoughtful design choice. The analyses of how earlier computations become components of later tasks, and why agents fail to reuse them, are insightful.

Critical weaknesses
  • Benchmark correctness / ceiling performance: Appendix C.3.2 identifies a significant flaw in the main evaluation worldline: at age 3, target outputs can depend on hidden random values that cannot be inferred from the input, affecting 17/20 problems. Other generated worlds repeatedly introduce essentially the same concepts. These issues raise concerns about benchmark validity and suggest the need for stronger automatic verification or a baseline for the ceiling performance the benchmark can support.

  • Harness selection and the interpretation of library learning: The proposed library learning is essentially an additional LLM session that refactors accumulated code into reusable functions and checks earlier examples. It is unclear how much this differs from standard coding-agent memory management and refactoring. Meanwhile, recent evidence from OpenAI (2026) shows that reasoning retention and context compaction can substantially improve ARC-AGI-3 performance. This raises the possibility that compaction already performs some form of implicit knowledge consolidation. The authors should clarify whether their harnesses use these capabilities and compare against stronger coding-agent baselines with compaction and periodic refactoring / reasoning trace visibility in the next turn.

  • Why should the learned library consist of code rather than natural-language concepts? An interesting aspect of ARC-∞ is that its own Designer uses natural language to represent previously acquired concepts and propose new conceptual compositions. The Director maintains descriptions of earlier concepts and generates natural-language growth intents, which the Creator subsequently translates into executable operations. This suggests that natural language may itself be an effective representation for accumulated conceptual knowledge, potentially supporting more flexible abstraction and transfer than executable functions alone. Yet the learner-side library learning focuses exclusively on consolidating Python code. It would be valuable to compare libraries containing (1) natural-language descriptions of learned concepts and rules, (2) executable functions, and (3) both. Such a comparison would help clarify whether conceptual bootstrapping specifically benefits from executable program composition, or whether maintaining and reusing explicit conceptual representations is sufficient.

Ref: OpenAI (2026). How enabling two settings tripled our scores on the ARC-AGI-3 benchmark. https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores/

Questions for authors
  • Why choose code as the primary representation? Could natural-language concept descriptions, or a hybrid code–language library, support comparable or better bootstrapping?

  • Could compaction itself provide implicit knowledge consolidation, and how would the proposed library learning compare against a standard coding agent with compaction and periodic refactoring? e.g., the arc-code is not a good harness in this sense.

6 comments

We thank the reviewer for the detailed review. We ran both suggested comparisons. One compares library learning with refactoring and compaction, and the other compares code with natural-language memory. We also fixed the age-3 world and reran every harness. The fixed world, the new checks and the comparison with refactoring are in the revised PDF in red. Below, the main worldline is the one used for the learners in Sec. 4.2, and the second worldline is the one in Appendix F.

Before addressing each point, we would like to clarify the scope of the paper. Like other benchmark papers, it proposes a benchmark, and how to solve the benchmark is not its main focus. Our harnesses are reasonable baselines that show that the benchmark works as intended, and designing a state-of-the-art learner is not our goal. For example, the ARC-AGI paper (Chollet, 2019) proposes the benchmark without proposing the best solver. SWE-bench (Jimenez et al., 2024) likewise reported simple baselines, and stronger agents such as SWE-agent (Yang et al., 2024) came in later work. Building better learners is left to those who use the benchmark. Within this scope, we ran every comparison the reviewer requested.


W1-1. At age 3 of the main worldline, answers depend on a hidden value (17/20 problems)

This is a single case that we found and reported ourselves in Appendix C.3.2. We fixed it, reran every harness, and all three findings hold. The answers at age 3 depended on a socket color that the input never showed. The input now shows the color through an opening in the cover, which also fixes the two age-4 problems that reuse this rule. We reran all five harnesses from age 0 and report both results.

HarnessCorrect (/220), submittedCorrect, fixedAge 3 (/20), submitted → fixed
Basic95913 → 9
Basic w/o memory78764 → 13
Schema44712 → 8
Tycho77712 → 7
Basic w/ library1271506 → 17

Memory still helps (91 vs. 76), library learning is the most accurate and the cheapest (2.36M vs. 3.62M output tokens for Basic), and all harnesses still fail in the last worlds (at most 7/20 at ages 9–10). This case occurred only at age 3 of the main worldline. The generator's instructions already forbid hidden values, and its review tests for them, but the review missed this case at age 3 (Appendix C.3.2). Run automatically over every test problem of both evaluation worldlines, the same test finds no other hidden value, and we will make it an automatic step of the generator.


W1-2. Other generated worlds repeatedly introduce essentially the same concepts.

This refers to the worldline we reported as a failure case in Appendix C.3.1, and such repetition is rare. That worldline was not used for evaluation and does not affect any reported result. If generated worlds generally repeated earlier concepts, many worlds would add nothing new. In the 10 worldlines we grew to age 50, only 2 of the 500 worlds (0.4%) add no new function or only renamed copies of earlier ones. In the evaluation worldlines, 74% (main) and 70% (second) of the problems need an operation introduced in their own world. Therefore, the repetition in Appendix C.3.1 is a rare failure, not a general tendency.

W1-3 & W1-4. These issues suggest the need for stronger automatic verification or a baseline for the ceiling performance the benchmark can support.

We provide both. A solver given the true operations sets a baseline for the ceiling, and we added automatic checks over every test problem. To estimate the ceiling, we provided a solver with the world's true operations, its training pairs and the problem's three support pairs, but not the program or the answer. In this experiment, the solver is correct on 219/220 (main) and 211/220 (second) problems. This is what a current LLM reaches when it is given the true operations. A learner that acquires these operations from the training pairs could therefore reach at least this score with a current LLM. A more capable LLM or harness could compose the operations better, so we expect the ceiling to be almost perfect. The best harness reaches 150/220 on the main worldline, so the benchmark leaves clear room above current harnesses.

We also added three automatic checks over every test problem of both evaluation worldlines, and we will build them into the generator so that every new world is checked before it is accepted. For hidden values, changing the generator's random values with the input fixed changes no answer after the fix. For ambiguity, alternatives that fit all three supports but change the answer exist in only 8/220 (main) and 22/220 (second) problems (ages 0–10). For repetition, the previous world's operations reproduce 97 of the 99 examples of the repeated worlds in Appendix C.3.1, so such a world can be rejected automatically.


W2-1. How does library learning differ from standard memory management and refactoring?

Library learning differs from refactoring in its goal. We ran the comparison the reviewer suggested, and library learning scored higher in every setting.

Both run a second session over the kept code, so they look alike. They differ in their goal. Library learning is an established method in program learning (DreamCoder, Ellis et al., 2021; LILO, Grand et al., 2024). It builds functions from solved problems so that later problems can be solved by composing them. Refactoring instead changes the internal structure of code without changing its observable behavior (Fowler, 2018). Our step follows the goal of library learning. It replaces values that the learner memorized for each example with a computation from the input. For example, at age 5 the learner first stored the masks of each shape it had seen. Library learning replaced this table with a function that folds the edge cells of any shape into a mask (Appendix E.4). Refactoring has no reason to remove such a table, because the table already reproduces every example.

If the two were the same, they would score the same on ARC-∞. We added a second session after each world. It either refactors the code and notes or applies our library-learning instruction. Refactoring organizes the code, removes duplication, extracts helpers and runs regression checks. Both use the same evidence, budget and test interface.

Main worldline, correct at ages 0–5 (/120)RefactoringLibrary learning
Code and notes, new conversation for each world4954
Code and notes, one long conversation with compaction5088
Library learning keeps code only (the best format, see W3)4985

In other words, harnesses that keep memorized examples overfit to each world (Finding 2), and library learning helps because it turns these examples into rules. We do not claim that library learning is the best learner. ARC-∞ is built to tell such strategies apart, and this comparison shows that it does.

W2-2. Do the harnesses use reasoning retention and compaction?

Yes, the harnesses use both reasoning retention and compaction. All harnesses run on Codex, which keeps the model's reasoning across tool calls within a session. We set no truncation. Schema keeps one conversation across all worlds and compacted it 60 times.


W2 & Q2-3. Could compaction already consolidate knowledge? How does library learning compare with compaction and periodic refactoring?

Compaction alone does not consolidate knowledge, and on top of compaction, library learning helps far more than refactoring (88 vs. 50). If compaction already consolidated knowledge, Schema would not overfit. It still overfits each world (Sec. 4.2) and scores 71/220, against 91 for Basic. In the table above, the long conversation with compaction reaches 88/120 with library learning and 50/120 with periodic refactoring. Compaction shortens the conversation, and library learning changes the knowledge that the learner keeps, so the two work together.


W2-4. Comparison with stronger coding-agent baselines

Schema and Tycho are our stronger coding-agent baselines. Both report near-perfect results on the public ARC-AGI-3 games, and both are among the harnesses we evaluate (Sec. 4.2). If the reviewer suggests another harness, we would be glad to run it on ARC-∞. As we noted above, finding the strongest harness is left to those who use the benchmark.


W3 & Q1. Why code rather than natural language? Could natural-language or hybrid libraries do as well?

We tried all three, and code worked best. We ran the three libraries the reviewer suggested with the same library-learning step and budget, so only what is kept between worlds differs. The natural-language learner can still write and run code within a world and at test time.

Kept between worldsCorrect, ages 0–5 (/120)
Code85
Natural-language descriptions74
Both54

Code can be run, so it can be checked against examples and combined directly, while a description must be turned into code again in every new world. Recent agent work finds the same. Executable code serves as a better skill library and action format than text (Voyager, Wang et al., 2023; CodeAct, Wang et al., 2024). On ARC, writing programs is better than predicting answers directly at precise computation and at composing several concepts (Li et al., 2025). The Designer works the same way. The Director plans in natural language, and the Creator turns each plan into code.


We hope these results address the reviewer's concerns. We welcome any further questions or requests for additional experiments.

References

  • Ellis, K., et al. (2021). DreamCoder: Bootstrapping Inductive Program Synthesis with Wake-Sleep Library Learning. PLDI 2021. link
  • Fowler, M. (2018). Refactoring: Improving the Design of Existing Code (2nd ed.). Addison-Wesley. link
  • Grand, G., et al. (2024). LILO: Learning Interpretable Libraries by Compressing and Documenting Code. ICLR 2024. link
  • Jimenez, C. E., et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024. link
  • Li, W.-D., et al. (2025). Combining Induction and Transduction for Abstract Reasoning. ICLR 2025. link
  • Wang, G., et al. (2023). Voyager: An Open-Ended Embodied Agent with Large Language Models. link
  • Wang, X., et al. (2024). Executable Code Actions Elicit Better LLM Agents (CodeAct). ICML 2024. link
  • Yang, J., et al. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS 2024. link
Reviewer_pLTrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ARC-infinity, a framework in which LLM agents generate a growing sequence of ARC-style "worlds", each built on the hidden operations of earlier ones, to measure conceptual bootstrapping. In its multi-agent framework, a director agent plans the next world from ranked depth and breadth candidates (controlled by αd\alpha_d and αb\alpha_b), and a creator agent implements it and generates training pairs and test problems. The authors show that the two α\alpha hyperparameters steer growth and ablate their design choices, then evaluate four agent harnesses on a 11-world worldline and find that memory helps, that all harnesses fail in later worlds, and that a library learning step helps.

Critical strengths
  1. The ablation of the generator is clear: each variant changes one component, and removing the candidate ranking, merging the two agents, giving the director access to the code, or limiting the history all make the library either stop diversifying or overwrite earlier operations. The one-shot comparison also makes a good case for growing the worlds step by step.

  2. I found the analysis of what the agents actually store interesting. Their saved code often ends up as lookup tables of training examples instead of general rules, which the paper uses to explain why the agents degrade in later worlds.

Critical weaknesses
  1. The paper does not show that the generated problems are unambiguously solvable: solvability is checked only by the LLM reviews inside the generator. Later worlds are more likely to contain ambiguous problems because each problem gives only three example pairs (and the world's 10 training pairs) while the library grows to 100+ operations. So when the harnesses perform poorly in the last worlds, it is unclear whether they failed to conceptual bootstrap or the problems could not be solved from the generated examples alone. Overall, as the code needed to solve each task grows more complex, it is unclear how later worlds remain solvable, which contradicts the arc-"infinity" in the title.

  2. The agent results come from a single worldline and don't hold up on the second one in the appendix. Memory makes almost no difference there, and the ranking of the agents changes, for example Tycho beats the library learning agent. Each agent is also run only once, and the same model generates the worlds and runs the agents. If the ranking changes from one worldline to the next, it's hard to use this as a measurement. I'd also like to see some evidence that doing well here relates to continual learning anywhere else.

Questions for authors
  1. How solvable are the problems at each age? For example can humans solve a sample, or can a solver with the true operation library solve them?
  2. Can you run the agents on more worldlines, with repeated runs, and explain how findings 1 and 3 hold up given the second worldline?
  3. Is there any evidence that performance on ARC-infinity correlates with performance on other continual learning or long-horizon tasks? While it's an interesting benchmark, I hope to understand its practical value.
6 comments

We thank the reviewer for the positive and constructive review. We ran both suggested experiments, a solver with the true operation library and repeated runs, and added a third worldline. These results are in the revised PDF in red. Below, the main worldline is the one used for the learners in Sec. 4.2, and the second worldline is the one in Appendix F.


W1 & Q1-1. Can the problems of each age be solved from the given examples? Can a solver with the true operation library solve them?

Yes, the problems can be solved from the given examples. A solver with the true operation library is correct on 219/220 (main) and 211/220 (second) problems, so the late failures of the harnesses come from what they learned.

To separate a learner's failure from an unsolvable problem, we ran a solver with the world's true operation library. If problems could not be solved from the given examples, this solver would also fail. It uses the same model as the learners and sees the world's operations, its training pairs and the problem's three support pairs, but not the program or the answer.

Solver with the true operation libraryMainSecond
Correct, ages 0–10219/220211/220
Lowest score at any age19/2017/20

At age 9 of the main worldline, the harnesses are correct on 0–3 of the 20 problems, and the solver on all 20.

We inspected the solver's 10 misses. Each wrong program is correct on all three support pairs and fails only on the test input. In 5 of the misses, the solver wrote its own version of a given operation instead of calling it, and this version differs from the true operation on the test input. In the other 5, the solver read part of the test input wrongly or chose another rule that also fits the support pairs. In other words, the solver had every operation it needed and made mistakes in applying them.


W1-2 & W1-3. Do later worlds contain more ambiguous problems and explain the late failures? How do later worlds remain solvable as the code grows?

Ambiguous problems do not necessarily increase as the library grows, and they are far too rare to explain the late failures. The late problems are hard, not unsolvable. We call a problem ambiguous if two programs reproduce all three support pairs but give different answers to the test input. We searched for such programs by skipping or replacing one operation of the true program. If a growing library made late problems ambiguous, many problems of the late worlds would be ambiguous. At ages 9 and 10 of the main worldline, at most 3 of the 20 problems of each age are ambiguous. Every harness fails on at least 10 of the same 20 problems, so ambiguity can explain at most 3 of these failures. The solver with the true operation library answers 39 of the 40 problems of ages 9 and 10.

The late problems are hard because their programs combine more operations from earlier worlds. On the main worldline, the program of a problem runs a median of 12 operations at age 1 and 26 at age 10. Section 4.2 (Finding 2) and Appendices C.1.4 and E.3 analyze why the harnesses fail on them. As the reviewer also noted, their saved code often ends up as lookup tables of training examples instead of general rules, so later worlds cannot build on it.

For a learner of fixed ability, such as a current LLM, later worlds become harder to solve, and we already observe this by age 10. For a learner whose ability grows as the worlds grow, we see no design flaw in ARC-∞ that would make the problems unsolvable. Such a flaw would appear if ambiguity grew with the library, but that is not the case. On the main worldline, the library grows from 79 functions at age 1 to 277 at age 10, yet at most 3 of the 20 problems of any age are ambiguous. On the second worldline, ambiguity is highest at age 3 (5/20) and is 0/20 at age 10.

W2 & Q2-1. Do Findings 1 and 3 hold across repeated runs and other worldlines?

Yes, Findings 1 and 3 point the same way on all three worldlines. If the findings were an artifact of one worldline or one run, they would flip on a repeat or on another worldline. We repeated the runs on the main worldline and added a third worldline. Finding 1 compares Basic with and without memory, and Finding 3 compares Basic with and without library learning.

Main, mean of 2 runsSecond (Appendix F)Third (new)
Basic vs Basic w/o memory, correct98.5 vs 79 (/220)91 vs 90 (/220)190 vs 171 (/220)
Output tokens3.50M vs 4.18M3.50M vs 3.93M2.17M vs 1.70M
Basic w/ library vs Basic, correct112 vs 98.5 (/220)97 vs 91 (/220)196 vs 190 (/220)
Output tokens2.79M vs 3.50M3.38M vs 3.50M1.32M vs 2.17M

Memory raises accuracy on all three worldlines and lowers cost on two. Library learning raises accuracy and lowers cost on all three. The gains are smaller on the second worldline, where fewer problems depend on earlier worlds (129/200 vs. 179/200).


W2-2. The ranking of the harnesses changes between worldlines. Can ARC-∞ still serve as a measurement?

Yes, ARC-∞ can still serve as a measurement. A ranking that changes between worldlines shows that no current harness dominates on every worldline, not that the worldlines are flawed. ARC-∞ evaluates a learner on several worldlines, as LLMs are evaluated on several benchmarks. LLM rankings also differ from one benchmark to another, and a model that does well on most of them is regarded as stronger. In the same way, a learner that does well on most worldlines is the stronger learner on ARC-∞. Measured against their own ablations, memory and library learning help on all three worldlines (W2 & Q2-1). A solver given the world's true operations answers 219 of the 220 problems of the main worldline. Every harness answers far fewer, so better learning methods have much room to improve.


W2-3. The same model generates the worlds and runs the agents

We agree this should be checked, and the runs are under way. We are running Basic, Basic w/o memory and Basic w/ library on the same worlds of the main worldline with several models other than GPT-5.6, the model that generated the worlds. We will post the results in this thread as soon as they are complete.

Q3. Does performance on ARC-∞ correlate with other continual-learning or long-horizon tasks, and what is its practical value?

We thank the reviewer for this interesting suggestion. Measuring the correlation needs the same learners run on several other benchmarks, which we could not complete within the rebuttal period, but we expect a correlation. ARC-∞ is itself a continual-learning and long-horizon task. For learners that update their parameters, each world is a new task that must not erase earlier ones. A worldline can also grow without bound, which makes it a long-horizon task. The components that help on ARC-∞ also help elsewhere. Library learning helps program learners in many domains (DreamCoder, Ellis et al., 2021; LILO, Grand et al., 2024), and a library of learned skills helps an agent in an open-ended game (Voyager, Wang et al., 2023). Our next work focuses on learners, and thanks to the reviewer's suggestion, we plan to measure this correlation there. If the results come early, we will add them to the camera-ready version.

The practical value of ARC-∞ is like that of ARC-AGI. ARC-AGI has become a major benchmark on which frontier LLMs are evaluated and compared (Chollet et al., 2025). Its value does not come from the puzzles as an application. It comes from the intelligence needed to solve them, and the community accepts ARC-AGI as a measure of this intelligence. ARC-∞ measures the same kind of intelligence and extends it to concepts that build on one another over time. To our knowledge, it is the first benchmark in which each task is built from the concepts of earlier ones. New worlds can always be generated, so the benchmark does not saturate.


We thank the reviewer again for the positive review. We welcome any further questions or requests for additional experiments, and we hope these results strengthen the reviewer's assessment.

References

  • Chollet, F. (2019). On the Measure of Intelligence. arXiv:1911.01547. link
  • Chollet, F., Knoop, M., Kamradt, G., Landers, B., & Pinkard, H. (2025). ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems. arXiv:2505.11831. link
  • Ellis, K., et al. (2021). DreamCoder: Bootstrapping Inductive Program Synthesis with Wake-Sleep Library Learning. PLDI 2021. link
  • Grand, G., et al. (2024). LILO: Learning Interpretable Libraries by Compressing and Documenting Code. ICLR 2024. link
  • Wang, G., et al. (2023). Voyager: An Open-Ended Embodied Agent with Large Language Models. link
Reviewer_vqxsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ARC-∞, a framework for automatically generating sequences of ARC-style worlds to evaluate conceptual bootstrapping without human involvement. The designer is split into a Director and a Creator: A Director proposes how new worlds should build on previously introduced concepts, while a Creator implements the corresponding operations and generates training and test problems over the depth and breadth of growth. In evaluation, there are depth and breadth controls and several design ablations, and the paper also compares agent harnesses on generated worldlines to study knowledge accumulation and reuse.

Critical strengths
  1. This paper's problem formulation is interesting: generating tasks with explicit dependencies on previously introduced concepts is meaningful for continual reasoning, especially for the evaluation of conceptual bootstrapping. The separation between the Director and Creator is also well motivated, decoupling the complex workflow of growing worldlines. There are also some examples showing interesting recursive compositions.
  2. The analysis of knowledge reuse/accumulation and failure modes is useful. Besides the final accuracy, the paper also counts how many new compositions are introduced along the worldline growth and how many are retained in the library of the final world. Those fine-grained analysis is useful to further evaluate the design of this workflow.
Critical weaknesses
  1. The quality and conceptual validity are not sufficiently evaluated. The main metrics used in the paper, Growth and Diversity, are based on code structure and code embeddings rather than the conceptual dependencies required to solve the generated problems. For example, cumulative growth can increase when successive worlds independently extend earlier computations without forming a progressively deeper cross-world dependency chain. It remains unclear how often the generated problems actually require the intended new concepts. Diversity is also a code-level embedding measurement, I think there is some missing validity check for the embedding model, and there should be some comparison between this embedding model and other modeling approaches. This matters because recursive conceptual growth is this paper's central contribution.
  2. The evaluation does not fully isolate the design choices of Director and Creator. The paper's current ablations change multiple factors simultaneously. For example, removing the Director also removes candidate ranking and the proposal stage, while substantially changing generation cost. The Director or the Creator has multiple components inside, so for a comprehensive ablation study, those components should also be isolated.
Questions for authors
  1. Can the authors provide a task-level audit or human annotation of the generated problems, including the fraction that require newly introduced cross-world conceptual dependencies? For example, does selectively removing the relevant prerequisite operations affect the outputs or learner performance? How bad it can affect the outputs? How frequently do ambiguous or redundant tasks occur? Those manual evaluation can help justify the validity of the problem generation method.
  2. How robust are the depth/breadth controls across independent generation runs with the same parameter? For example, will the same-rank candidates in multiple generation runs have similar "concept depths"? The rigor of these depth and breadth metrics needs further elaboration, since they are the important tunable parameters of the problem generation workflow.
8 comments

We thank the reviewer for the constructive review. We ran the suggested task-level audit on every test problem and added its results to the revised PDF in red. Below, the main worldline is the one used for the learners in Sec. 4.2, and the second worldline is the one in Appendix F.


W1 & Q1-1. Do the generated problems require the newly introduced concepts and the earlier ones?

Yes, the generated problems need both the new concepts and the earlier ones. On the main worldline, 74% of the problems need an operation introduced in their own world, and 90% need an operation from an earlier world.

To see whether a problem needs an operation, we removed the operation and checked whether the answer changes. If the problem did not need the operation, its answer would stay the same. If the problem depends on it, the answer would change. We did this for every operation of every test problem, with the test input fixed. A removed operation returns its input unchanged or a typical output of that operation. The table counts the problems whose answer changes.

Problems of ages 1–10 whose answer changes when one operation is removedMainSecond
An operation introduced in the problem's own world148/200 (74%)140/200 (70%)
An operation from an earlier world179/200 (90%)129/200 (65%)

This suggests that most generated problems cannot be solved without the concepts of their own world and of earlier worlds. A learner that has not acquired the earlier concepts cannot solve 90% of the problems on the main worldline.

The quarter of the problems that does not need its own world's operation reflects a design choice. We separated the process that introduces concepts from the process that builds problems. A new operation enters the library and problems are built from the whole library, so we did not force every problem to use the newest operation. These problems instead compose earlier operations in a new and more complex way, which is another way for a world to grow. A new operation that is not used in its own world stays in the library and is available to the problems of later worlds. In the current setting about three quarters of the problems use the newest operation, and this share can be raised by requiring it in every problem.


W1 & Q1-2. Does removing a prerequisite operation affect learner performance, and how much?

Yes, removing the operations of the new world lowers the accuracy of a solver on both worldlines.

To see the effect on a learner, we gave a solver the world's training pairs, the problem's three support pairs and a library of the true operations, and compared two libraries. If the new world's operations were not needed, the solver would score the same with both.

Operation library given to the solverMain, ages 1–4Second, ages 1–4
All worlds up to the problem's own world80/8080/80
Earlier worlds only53/8064/80

Without the new world's operations, the solver loses 27 problems (34%) on the main worldline and 16 (20%) on the second. For the outputs, every change counted in the first table turns a correct answer into a wrong one, since an answer counts only if every cell matches. This suggests that a solver needs the operations of the new world to answer these problems.

W1-3. Do the worlds form progressively deeper chains, or do they extend earlier computations independently?

They form deeper chains, so the growth is recursive. In the 10 worldlines we grew to age 50, 74% of the worlds of ages 1–10 lie on a chain of three or more worlds, and this rises to 80% over ages 1–50.

To see this, we followed the calls of each world's new operations back through earlier worlds and counted the worlds on the longest chain. If worlds only extended earlier computations independently, new operations would call earlier ones directly, and chains would stop at two worlds. A chain of three or more worlds means that a world builds on an earlier world that itself built on an even earlier one, which is the recursive growth the reviewer refers to. We grew 10 worldlines to age 50 and measured them at ages 10, 30 and 50.

Longest chain of a world's new operations (10 worldlines)Ages 1–10 (100 worlds)Ages 1–30 (300)Ages 1–50 (500)
Calls no earlier world4%3%2%
2 worlds22%17%18%
3 worlds32%34%30%
4 worlds25%17%16%
5 worlds16%13%16%
6 worlds or more1%15%18%

Over ages 1–10, 74% of the worlds lie on a chain of three or more worlds. Over ages 1–50, this rises to 80%, and 34% lie on a chain of five or more. The longest chain in a worldline spans 4.3 ± 1.2 worlds at age 10 and 7.5 ± 5.0 at age 50 (mean ± s.d. over the 10 worldlines), and the longest we found spans 21 worlds.


W1-4. Is the Diversity embedding valid, and how does it compare with other measures?

The conclusions do not depend on a single embedding model. If they were an artifact of CodeRankEmbed, another encoder or a measure without a learned model would order the conditions differently. We therefore recomputed Diversity with two other measures. One is another learned code encoder, and the other uses no learned model. UniXcoder (Guo et al., 2022) is trained on different data with different objectives. It was evaluated on zero-shot code-to-code search, which compares code fragments by their embeddings as Diversity does. AST features use no learned model. They describe code by counting syntax-tree patterns, as tree-based clone detection does (DECKARD, Jiang et al., 2007). Our features count the kinds of syntax-tree nodes (such as loops, calls and comparisons), the pairs of a node and the node directly inside it, and the small patterns that a node forms with its children and grandchildren. Names, comments and constants are left out.

Diversity gain at age 10 (runs of Table 1)CodeRankEmbed (paper)UniXcoderAST features
Full+3.27+1.04+3.15
w/o Candidate search+0.09−7.17+2.34
w/o Director−0.08−4.21+1.96

Full stays above both variants under all three measures, and higher breadth raises Diversity under all three. Over 220 generated worlds, the rank correlation with our measure is 0.83 for UniXcoder and 0.47 for AST features. Therefore, the Diversity results reflect differences in the generated code, not a property of one embedding model.

Q1-3. How often are problems ambiguous?

Few problems are ambiguous. We keep a small amount of ambiguity so that the benchmark tests learning from a few examples, and a solver given the true operations resolves almost all of it. We call a problem ambiguous if two different programs reproduce all three support pairs but give different answers to the test input. To find such problems, we made alternatives to the true program of every problem by skipping one operation or replacing it with another operation that takes and returns the same types. Only 8/220 problems (4%) of the main worldline and 22/220 (10%) of the second have such an alternative (ages 0–10).

Ambiguity depends on the number of support pairs. A few examples seldom determine a program uniquely (Mitchell, 1980; Ellis & Gulwani, 2017), and with more support pairs, fewer programs fit all of them. Removing ambiguity entirely, however, would need many support pairs per problem. A learner could then find each rule from these pairs alone, without reusing the concepts it acquired in earlier worlds, and the benchmark would no longer measure learning efficiency. Like ARC-AGI, ARC-∞ measures how efficiently a learner learns a rule from a few examples (Chollet, 2019), and it asks the learner to do so by reusing operations it learned in earlier worlds. We therefore keep three support pairs and accept a small amount of ambiguity.

We expect a learner to resolve the remaining ambiguity with priors. ARC-AGI also relies on priors close to human ones, such as preferring the rule with the shortest description (Chollet, 2019; Feldman, 2000). In ARC-∞, we expect a learner to use the operations it has acquired from the training pairs as a prior. We also expect it to use its own prior to some extent, such as the knowledge of its underlying LLM. To test this, we gave a solver the true operations, the world's training pairs and the problem's three support pairs. If these could not resolve the ambiguity, the solver would fail on the ambiguous problems. It answers 219/220 problems of the main worldline and 211/220 of the second (ages 0–10). This suggests that the operations a learner acquires resolve almost all of the ambiguity.


Q1-4. How often are problems redundant?

No test problem repeats another. We call two problems redundant if they apply the same task program to the same query input. Solving one then gives the answer to the other. Formally, problems ii and jj are redundant if zi=zjz_i = z_j and xi∗=xj∗x^*_i = x^*_j, where zz is the task program and x∗x^* the query input (Sec. 2). We checked all 3,300 test problems of nine worldlines (ages 0–10). Among them, no two problems are redundant, and no test pair repeats one of its own support pairs or a training pair. Problems of a world do share rules by design. Each world tests a few rules, and each rule is tested on several different inputs (Appendix A.4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.