acceptodds
Under review as a conference paper at ICLR 2027

Active Exploration, Rule Discovery, and Teaching in ARC

Abstract

Abstract reasoning benchmarks such as the Abstraction and Reasoning Corpus (ARC) typically provide models with a small set of demonstrations and evaluate whether they can infer the underlying transformation rule. However, the selection of these demonstrations and its role in successful inferences has received comparatively little attention. In this work, we instantiate interactive environments for ARC tasks using task-specific generator and verifier programs, allowing us to study both active discovery and teaching. We introduce ActiveARC, a framework in which models can construct input grids, observe their transformations through an oracle, and then solve held-out instances. We further ask whether models that already know the underlying rule can select demonstrations that effectively teach it to another model. Across four ARC-style datasets and five model families, requiring models to actively select their own examples substantially reduces performance relative to randomly sampled examples under matched query budgets. Accuracy drops by 12.8 percentage points for official items on average and 13.6 points for freshly sampled items, with active selection underperforming random sampling in all evaluated settings. We find that models systematically construct smaller experiments than both human-authored training examples and random samples from the task generator, and in most settings less diverse ones. Surprisingly, human-authored demonstrations often provide only a modest advantage over random generator samples, especially on freshly sampled items. In the teaching setting, when given explicit knowledge of the transformation, the strongest model generates demonstrations that teach other models nearly as effectively as human-authored examples. The other models teach considerably less effectively, even when they can apply the provided rules themselves. With an unconstrained interaction budget, most models do not discover substantially better and typically stop after only a few queries. Our results suggest that inferring a rule from given examples and gathering the right examples oneself are distinct capabilities, and that current models may lack the latter even when they have the former.

Primary area:
datasets and benchmarks
Keywords:
abstract reasoning, active learning, machine teaching, interactive reasoning, ARC

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wAHhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ActiveARC, an interactive environments for abstract reasoning benchmarks like ARC using generator and verifier programs. This could lead to whether models can choose good examples by themselves, not only infer a rule from given ones. On four ARC-style datasets with five LLMs, self-selected queries are worse than random generator samples.

Critical strengths
  1. The setup is simple, straightforward and interesting. Using generator and verifier programs to make ARC interactive is a very interesting idea, and comparing Active-K with Random-K at exactly the same number of pairs is a clean design.

  2. Methods are compared cleanly. Active-K and Random-K get exactly the same number of pair, and Active-K is lower in all 20 cell.

  3. There are many robustness checks included in the paper, like abandoned trials, a larger budget, noisy outputs and slippage.

Critical weaknesses
  1. Only compare the model with Random-K baseline is not fair. It samples from the hand-written generator, which already encodes the factors of variation of each task, but the model has no such knowledge.

  2. The teaching claim relies on saturated benchmarks. Static is 93-99.5% on ARC-AGI-1/2 for the top models, so contamination is a concern.

  3. All main runs use low effort, and Gemini answers 36-84% of exam items without reasoning.

Questions for authors
  1. I'm wondering if it is possible to add a baseline with random valid grids or with simple perturbations of the initial sample? This would show how much of the Random-K advantage comes from the generator's knowledge.

  2. How do the main results change with medium or high reasoning effort for more than one model?

3 comments

We thank the reviewer for the positive assessment of the design and for the three concrete concerns. We addressed each of them with new experiments or further analyses of the existing runs, which are all part of the revised manuscript (mainly Sec. 3.3, Sec. 4.2, Sec. 4.4 and Appendix A.7 to A.17).

W1, Q1: Is random-K a fair baseline, and how do baselines without the generator's knowledge perform?

▶\blacktriangleright It is correct that the generator's inputs follow the input distribution of the task, which the model does not know in advance. We still think that random-K is the appropriate reference, for the following reason. The generator provides valid inputs, but these are drawn at random. Random-K is in that sense an uninformed selection, since the generator does not know which rules the model is considering and never sees an output. The model, however, sees the output of each query before it chooses the next one, and can choose the input that separates the rules it is considering. Such an informed selection is expected to be better than a random one, which is why random sampling is the standard reference for active learning. The model, the number of pairs and the test items are identical in both settings. We now state this in Sec. 3.3: "[...] the only difference is whether the agent chose the inputs itself or whether they were sampled from the generator, whose inputs are valid but not selected to be informative, while the agent can adapt each query to earlier outputs."

We also added the two baselines that the reviewer suggests as a new experiment (Sec. 4.2, A.11). Both provide the model with valid inputs that require no knowledge of the task. Perturbations recolor 5 to 30% of the cells of the first example, or swap two of its colors. Random grids are uniformly random grids with the shape and colors of the first example. In both cases, only inputs that the verifier accepts are kept, and their outputs come from the same verifier. The table shows the random grids (accuracy on the official test items):

BenchmarkModelRandom gridsActive-KRandom-K
ARC-AGI-2GPT-6 Astra82.988.897.1
ARC-AGI-2Gemini 3.8 Flash68.475.985.6
ARC-AGI-2DeepSeek V4.1 Flash*71.161.891.4
ARC-AGI-2GPT-5.6 Luna33.727.845.7
P-ARCGPT-6 Astra61.175.077.8
P-ARCGemini 3.8 Flash52.644.760.5
P-ARCDeepSeek V4.1 Flash*35.015.042.5
P-ARCGPT-5.6 Luna0.05.68.3

*Abandoned Active-K trials count as incorrect; without them, random grids still lead.

The queries that the models choose themselves are not consistently better than inputs that are chosen without any knowledge of the task. Active-K is below the random grids in four of the eight cells (DeepSeek V4.1 Flash on both benchmarks, Gemini 3.8 Flash on P-ARC and GPT-5.6 Luna on ARC-AGI-2), and averaged over the four models on ARC-AGI-2, the models' queries are as useful as random valid grids (+0.5 points in favor of the random grids, 95% interval [-2.8, +3.8]). The perturbations give a similar picture (A.11). At the same time, the random grids are valid inputs as well, but remain below random-K in every cell. The advantage of random-K over Active-K can therefore not be explained by the generator knowing which inputs are valid. It rather depends on which inputs are observed, and random-K, as described above, does not select them in any way.

Three further results show that the gap is not an artifact of what the generator knows. First, random samples are far from the most informative evidence. On P-ARC, they rule out only 62.6% of the incorrect rules that are consistent with the first example (A.12), so that a query which targets the remaining ones would be more informative than a random sample. The queries of the models rule out 28.6%. Second, a model that knows the rule can exceed the generator. In the teaching setting, the model is told the rule and selects which inputs to show. The demonstrations of GPT-6 Astra are more useful for the students than random-K on three of four benchmarks (57.8, 80.4 and 87.5% compared to 52.0, 73.3 and 84.7% on P-ARC, ConceptARC and ARC-AGI-1; ARC-AGI-2: 81.4 compared to 82.8%; A.15). Third, providing the generator's inputs is not sufficient. In a new experiment, the model selects its inputs from a pool that contains its own earlier queries and the same number of generator inputs, without any indication of their source. Only 39% of its selections are generator inputs (A.13). Taken together, the gap reflects that the models do not select the inputs that discriminate between candidate rules, which is what the active setting is designed to measure.

[1/2]

W2: Does the teaching claim rely on saturated benchmarks, and is contamination a concern?

▶\blacktriangleright ARC-AGI-1 and ARC-AGI-2 are indeed close to saturated for the strongest students, which we note in the limitations. The teaching results do not depend on these cases, however. Each teaching result is a combination of a student and a benchmark, and of these 20 combinations, ten have a static accuracy below 90% (down to 10% for GPT-5.6 Luna on P-ARC). To show this, we split the existing results by the static accuracy of the student (A.16). The table gives the mean difference between a teacher's demonstrations and the human-authored ones (points, official / sampled test items):

TeacherAll (20)Static below 90% (10)Static below 80% (7)Static 90% or above (10)
GPT-6 Astra-1.1 / -1.2-0.2 / -1.9+0.4 / -2.2-1.9 / -0.6
Claude Opus 5-8.3 / -4.8-8.4 / -4.3-8.7 / -3.7-8.2 / -5.4
Gemini 3.8 Flash-8.9 / -7.2-8.0 / -8.1-8.0 / -9.2-9.8 / -6.2
DeepSeek V4.1 Flash-9.9 / -8.2-9.6 / -8.7-9.6 / -10.0-10.2 / -7.6
GPT-5.6 Luna-16.9 / -13.7-15.7 / -12.3-16.4 / -13.8-18.1 / -15.1

The results do not change when the saturated cases are excluded. GPT-6 Astra stays within about two points of the human-authored demonstrations in every subset, and all other teachers remain below them. The cases at 90% or above also differentiate between the teachers, with losses of 8 to 18 points for the four weaker ones, i.e., the students depend on the demonstrations there as well.

Regarding contamination, there are four reasons why we do not think that it explains the teaching results. First, the results are the same on the sampled test items, which are freshly drawn from the generator and cannot have been seen before (second number in each cell of the table). Second, the teachers do not reproduce the official examples. Of their demonstration inputs, none is identical to an official input of the same task on P-ARC and ConceptARC, and at most 1.6% are on ARC-AGI-1 and ARC-AGI-2 (A.15). Third, the VARC student of the submitted version is trained from scratch on the demonstrations of each task alone and cannot be contaminated. With this student, the ranking of the teachers closely mirrors the one obtained with LLM students (A.24). Fourth, familiarity with the official tasks would favor the static setting, since these are the demonstrations that a model may have seen during training. It would therefore make the teachers look worse and not better.

W3, Q2: Low reasoning effort, and Gemini answering test items without reasoning

▶\blacktriangleright We ran both settings at high reasoning effort for three further models as a new experiment (Sec. 4.4, A.7; one draw per cell). Accuracy on official / sampled test items:

BenchmarkModelLow effort Active-KLow effort Random-KHigh effort Active-KHigh effort Random-K
P-ARCGemini 3.8 Flash50 / 5455 / 6968 / 7676 / 92
P-ARCClaude Opus 547 / 5562 / 7372 / 7482 / 94
ARC-AGI-2GPT-5.6 Luna28 / 3947.2 / 56.556 / 6075.5 / 82

High effort raises the accuracy in both settings by 18 to 30 points, but Active-K remains below random-K in all three cases and on both item types. More reasoning seems to make the models better at inferring a rule from the evidence, but it does not help them to choose the evidence. We could not repeat all 20 combinations of model and benchmark at high effort due to the cost of these runs.

Regarding the teacher's own test items, the reviewer is right that Gemini 3.8 Flash skips the reasoning at low effort. Before teaching, each teacher has to solve the test inputs of the task itself, and at low effort Gemini answers most of them without any reasoning. In a new experiment, we therefore repeat its teaching runs on P-ARC and ARC-AGI-2 at high effort, with the students unchanged (A.17). The teacher then reasons on every test input and solves 97.0% (P-ARC) and 98.3% (ARC-AGI-2) of them, compared to 38.0% and 73.2% at low effort. Its demonstrations improve as well: on P-ARC, three of its four students are 8 to 16 points more accurate than with the low-effort teacher (GPT-6 Astra 86 compared to 74%, Gemini 3.8 Flash 68 compared to 60%, DeepSeek V4.1 Flash 58 compared to 42%), while the changes on ARC-AGI-2 are small. The reasoning effort does matter for the teaching quality of this model. Our observation that being able to apply a rule is not sufficient for teaching it does not depend on Gemini 3.8 Flash, however, since Claude Opus 5 and DeepSeek V4.1 Flash solve 81 to 94% of their own test items and still teach considerably worse than GPT-6 Astra.

We appreciate the reviewer's positive view of the setup and the three precise concerns. Each of them is now covered by an experiment or analysis in the revised manuscript, and we welcome any remaining questions.

[2/2]

Thanks for the author's detailed rebuttal! But I still have some concerns regarding the generalizability of the proposed method to broader datasets. Therefore I would like to keep my score.

Reviewer_LKAnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ActiveARC, a framework for studying how models discover and teach rules in ARC-style tasks. Across five models and four benchmarks, models perform worse with their own examples than with the same number of randomly generated examples. Their queries tend to be smaller and often less diverse, and most models stop after few queries when allowed to choose how many to make. In teaching, models receive the rule and construct examples for another model. The strongest teacher performs nearly as well as human-written examples for LLM students, but a gap remains for a student trained from scratch. These results suggest that learning from examples and choosing useful examples are distinct abilities.

Critical strengths
  1. ActiveARC offers a useful framework for studying how models acquire and communicate evidence, with a consistent finding that Active-K underperforms Random-K in all 20 model–benchmark combinations on both official and freshly sampled items, even after abandoned trials are excluded.

  2. The additional analyses make the findings more informative: evidence replay helps distinguish poor demonstration selection from poor use of the evidence, while cross-model teaching and the V_ARC experiments reveal how teaching effectiveness depends on the student’s prior knowledge.

  3. The paper is well written and experimentally thorough, covering five models and four benchmarks, with checks on query budgets, reasoning effort, misleading initial examples, and noisy observations. The clear presentation and detailed appendix make the methods and results easy to assess.

Critical weaknesses
  1. In Active-K, a model has to construct valid inputs after seeing just one demonstration. Random-K receives examples from a generator that already encodes the task’s input constraints. The performance gap may therefore reflect difficulties in constructing suitable inputs as well as choosing informative examples. The results clearly show that the current active setup performs worse, but they provide less direct evidence that models are poor at selecting useful evidence. Letting both conditions select from the same pool of valid inputs would help separate these abilities. This matters for how broadly the main finding can be interpreted.

  2. Many models make few queries in Active-Free, so this experiment tells us more about when they stop than about whether they can benefit from sustained exploration. Adding only two queries provides limited evidence on the latter question. The diversity analysis also measures properties such as input size and similarity without showing whether the queries distinguish possible rules or whether this explains task success. Prompting receives little attention, even though the reported changes in behavior suggest that instructions may matter. Together, these issues leave the reasons for the active–random gap insufficiently tested.

Questions for authors
  1. Could you clarify what evidence separates difficulties in constructing valid inputs from difficulties in choosing informative ones? A comparison in which both conditions see the same pool of valid, unlabeled inputs and receive outputs for the same number of selections would help address this concern.

  2. Do you have results with larger fixed query budgets? A performance curve across several budgets would help show whether the main problem is stopping too early or failing to benefit from additional queries.

  3. How sensitive are the results to the exploration instructions? Have you tested prompts that ask models to consider alternative rules and design queries that distinguish them? This would help assess whether the observed behavior is stable across reasonable prompting choices.

  4. Can you provide evidence that the queries cover variations relevant to the rule, and that this coverage is related to task success? This would help explain the performance gap beyond differences in input size or visual similarity.

3 comments

We thank the reviewer for the careful reading and the concrete suggestions. We ran the proposed experiments and added further analyses of the existing runs, which are all part of the revised manuscript (mainly Sec. 4.2, Sec. 4.4 and Appendix A.4 to A.14).

W1, Q1: Constructing valid inputs or selecting informative ones?

▶\blacktriangleright We agree that these two abilities have to be separated, and we ran the comparison that the reviewer describes as a new experiment (Sec. 4.2, A.13). For each task, we fix a pool of eight valid, unlabeled generator inputs. In Pool-K, the model sees the first example and the pool, selects one candidate per query and receives its output before the next selection, for the same number of queries as in Active-K. As a control, the same number of candidates is selected from the same pool at random, so that both conditions see the same pool and receive outputs for the same number of selections. Accuracy on the official test items (GPT-6 Astra and Gemini 3.8 Flash, two draws):

BenchmarkModelActive-KModel selectsRandom selectionDifference [95% interval]Random-K
P-ARCAstra72.083.081.0+2.0 [-7.0, +11.0]80.0
P-ARCGemini50.059.056.0+3.0 [-8.0, +14.0]55.0
ARC-AGI-2Astra88.999.597.5+2.0 [+0.5, +3.8]96.7
ARC-AGI-2Gemini76.687.482.9+4.5 [-0.5, +9.5]86.2

Without the construction step, the models' selections are only marginally better than random selections from the same pool (2.0 to 4.5 points). That the accuracy rises to the level of random-K is expected, since random-K is itself just an uninformed selection of generator inputs, so that a selection from a pool of generator inputs cannot be worse than it. The comparison mainly shows that a selection by the model adds very little over a random one.

Selecting when the candidates differ. To test the selection where it matters, we ran a second new experiment with a mixed pool. It contains the model's own queries from an earlier Active-K trial on the same task and the same number of generator inputs, in random order and without any indication of their source. If the model selects only its own queries, it ends up with its Active-K evidence, and if it selects only the generator inputs, with the evidence of random-K. A model that recognized the more informative candidates could therefore reach the level of random-K in every trial. Accuracy averaged over official and sampled test items:

BenchmarkModelOwn queries only (Active-K)Model's selectionGenerator inputs only (random-K)Selection minus random-K [95% interval]
P-ARCAstra80.684.785.2-0.5 [-8.7, +7.1]
P-ARCGemini51.245.861.9-16.1 [-26.8, -5.4]
ARC-AGI-2Astra89.397.597.8-0.4 [-2.5, +1.5]
ARC-AGI-2Gemini78.180.886.3-5.5 [-10.1, -1.1]

Pooled over both models and benchmarks, only 39% of the selections are generator inputs (95% interval [36.5, 42.4]), i.e., the models tend to keep their own queries. Their selections do not exceed random-K in any of the four cells and remain 3.9 points below it when pooled ([-6.2, -1.5]). Gemini 3.8 Flash stays considerably below this level (-7.4 points, [-11.5, -3.2]), whereas GPT-6 Astra reaches it once generator inputs are part of its evidence.

Producing valid inputs. Two further results show that validity is not what separates the two settings. First, we split the existing Active-K trials and only keep those in which no query was rejected. Compared to random-K on the same tasks, the gap is then -10.6 / -12.3 points (official / sampled items), compared to -12.8 / -13.6 points over all trials (A.9). Second, in a new experiment we replace the queries of a model by random grids that have the shape and colors of the first example and that the verifier accepts, i.e., valid inputs that require no understanding of the task. On ARC-AGI-2, these are on average as useful as the queries that the models choose themselves (+0.5 points in favor of the random grids, [-2.8, +3.8]), and both are considerably less useful than generator samples (A.11).

We summarize these results in Sec. 4.2 as follows: "Taken together, the models are able to infer a rule from informative evidence and to produce valid inputs, but do not seem to identify the inputs on which candidate rules differ, whether they construct or select them."

[1/3]

W2, Q2: Do models stop too early, or do they fail to benefit from additional queries?

▶\blacktriangleright We agree that the Active-Free setting mixes these two questions. Our main comparison is however Active-K, in which the number of queries is fixed, so that the model cannot stop early. To cover larger budgets as well, we ran a new experiment in which both settings receive two, four and eight additional pairs (Sec. 4.4, A.4). We use P-ARC for this, since it is the benchmark that is least saturated in the passive settings, so that additional evidence can help the most, and since its size allows us to run three larger budgets with two draws each. GPT-6 Astra and Gemini 3.8 Flash are the two models with the smallest gap at the regular budget. Accuracy on official / sampled test items:

KK+2K+4K+8
Astra, Active-K72 / 8377 / 8572 / 8582 / 86
Astra, random-K80 / 9089 / 9888 / 9487 / 95
Gemini, Active-K50 / 5448 / 5237 / 4745 / 51
Gemini, random-K55 / 6955 / 6563 / 6658 / 68

Active-K is below random-K at every budget and on both item types. Averaged over the three larger budgets, the difference is -11.0 points for GPT-6 Astra (95% interval [-18.0, -4.3]) and -15.3 points for Gemini 3.8 Flash ([-23.7, -7.0]) on the official items, and -10.3 and -16.3 points on the sampled items. The additional queries also barely change the accuracy of the active setting. Even with eight more queries, Active-K does not reach what random-K achieves with the regular budget (first column), with GPT-6 Astra on the official items as the only exception (82 compared to 80%). This suggests that the main problem is not that the models stop too early, but that their additional queries add little information. The appendix also shows these results as a curve (Figure 6).

W2, Q3: How sensitive are the results to the exploration instructions?

▶\blacktriangleright We tested the instruction that the reviewer suggests in a new experiment (Sec. 4.4, A.14). The following text is added to the Active-K prompt: "Before each query, write down at least two different rules that are consistent with everything you have seen so far. Then choose an input grid on which these rules would produce different outputs, so that the result tells you which of them is correct." Accuracy on P-ARC (official / sampled test items, two draws):

ModelActive-KWith instructionRandom-K
GPT-6 Astra72 / 8363 / 7780 / 90
Gemini 3.8 Flash50 / 5445 / 6455 / 69
Claude Opus 547 / 5543 / 5662 / 73

The models follow the instruction and write down candidate rules before their queries, but all three remain below random-K on both item types. Compared to the regular prompt, the accuracy on the official items is lower for all three models, and the effect on the sampled items is mixed. The gap can therefore not be attributed to a prompt that fails to suggest a strategy.

We agree with the reviewer that instructions matter for how the models explore. In the program-induction variant of the submitted version (A.6), GPT-6 Astra and Claude Opus 5 query about twice as often when they have to submit a program instead of an output grid. They do however not seem to change how informative the queries are, since neither more queries (see Q2), nor an explicit strategy, nor more reasoning closes the gap. For the latter, we ran both settings at high instead of low reasoning effort for three models. Accuracy increases by 18 to 30 points in both settings, but Active-K remains below random-K in all three cases (A.7). In smaller pilot experiments with individual models (20 to 30 tasks each), we additionally required the model to follow a fixed query protocol: to predict the output of each query before submitting it, to first list the factors that vary between the inputs of a task and then cover different combinations of them, or to change only one aspect of an already observed input per query. None of these protocols improved the accuracy compared to the regular setting. In the main experiments we tested one alternative instruction, which we now state in the limitations.

[2/3]

W2, Q4: Do the queries cover variations relevant to the rule, and is this related to task success?

▶\blacktriangleright We agree that the analysis in the submitted version described what the queries look like (size, similarity), but not what they test. The revised Sec. 4.2 now measures the latter directly on the existing trials (A.12). A query covers a variation that is relevant to the rule if its output distinguishes the true rule from another rule that is still plausible. As such competing rules we use the incorrect programs that models submit when they are asked to write down the rule as code, restricted to those that agree with the true rule on the first example of a trial. We then count how many of them are contradicted by the further evidence of that trial:

Competing rules ruled out (%)P-ARCConceptARCARC-AGI-1ARC-AGI-2
Queries chosen by the model (Active-K)28.627.653.851.8
Random generator samples (random-K)62.636.965.680.8

As a second set of competing rules, we use near-miss variants of the true rule. For ARC-AGI-1 tasks with a compact reference program, we remove one operation at a time and keep the resulting rules that still agree with the true rule on the first example. The queries of the models rule out 50.8% of these near-miss rules, compared to 69.8% for random samples. In both cases, the evidence that models gather for themselves rules out considerably fewer incorrect rules than random evidence of the same size, and this measure does not depend on the size or similarity of the inputs.

That this is related to task success follows from three comparisons in which only the evidence changes. First, Active-K and random-K use the same model, the same number of pairs and the same test items, and differ only in which inputs are observed. Second, in the evidence replay of the submitted version (A.10), GPT-6 Astra answers the P-ARC test items from the evidence that other models gathered. Their queries change its accuracy by at most three points compared to the first example alone, whereas the same number of generator samples brings it to 80%. Third, if the same model selects from generator inputs instead of constructing its own inputs, its accuracy rises to the level of random-K (see Q1).

The reviewer's point about size and similarity is also supported by a pilot experiment. If we prevent DeepSeek V4.1 Flash from submitting near-copies of earlier inputs and inputs of the same size on 20 ConceptARC tasks, the diversity of its evidence rises to the level of the official examples, but its accuracy does not change (10 compared to 9 solved tasks). Size and similarity describe how the queries differ from random samples, but for solving the task it seems to be more important whether the queries vary what the rule depends on.

Overall, the reviewer's suggestions made the paper considerably more specific about why active evidence falls short, and we thank them for that. The revised version no longer only reports the gap between Active-K and random-K, but also tests where it comes from, with the experiments proposed in the review. If any of the points above remain unclear, or if further comparisons would be helpful, we are glad to provide them during the discussion period.

[3/3]

Reviewer_2ZNfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ActiveARC, an interactive framework that converts static ARC tasks into dynamic environments to evaluate frontier LLMs' active discovery and teaching capabilities. When models actively queried and collected their own training examples, their accuracy dropped by 12.8–13.6%p compared to receiving random samples under identical conditions. It is observed that the models generate queries with smaller grid sizes and limited visual diversity, and fail to actively query even when given unrestricted budgets. In contrast, when models created teaching examples for other AI, only the best model (GPT-6 Astra) made examples as good as humans did. Ultimately, the study demonstrates that inferring rules from provided information and actively selecting optimal evidence to discover rules are fundamentally distinct intelligence capabilities.

Critical strengths

Existing ARC problems mainly evaluate the ability to infer a rule from the structure of input-output pairs. In this work, the authors built an environment where, given multiple candidate hypotheses, the model can test a new input against each hypothesis and verify for itself whether the result is logically consistent. Other than active discovery, they also designed the setup so that teaching ability can be tested as well, by having the model produce inputs for another model to learn from.

Critical weaknesses
  1. The paper operates primarily as an empirical observation report rather than proposing concrete methods to resolve LLMs' active exploration failures. Furthermore, while ActiveARC is presented as a major contribution, the study lacks demonstrations on how this environment can be leveraged to train or help models overcome these limitations, diminishing its overall practical impact.

  2. Defining "teaching" merely as generating static demonstration pairs represents an overly simplistic view of pedagogy that excludes interactive guidance and corrective feedback. This narrow framing raises fundamental questions about whether teacher models are genuinely imparting rule understanding or simply synthesizing additional data points.

  3. Concerns about the execution: However, LLMs do not actually perform active discovery well in this setting. They fail to generate genuinely informative inputs, and performance does not improve significantly. Moreover, comparing Active-K against Random-K alone makes it hard to tell which capability the model is lacking. The teaching component is also questionable: in practice, another model is just producing inputs that are not in the example pairs, and it is unclear whether this qualifies as teaching.

Questions for authors
  • What is the real purpose of newly evaluating frontier LLMs, which already solve static ARC well, in an interactive environment? If the goal is to see whether models explore actively, I think we should look for a “breakthrough” result, where the teaching experiment actually enables a model that cannot solve a problem to solve it. Since the reported performance shows only drops, the paper could be framed as exposing the raw weaknesses of frontier LLMs’ problem solving, but that would require much more analysis.
  • Isn’t the fact that the models minimized queries and submitted only small, biased grids evidence that they solved the problems through pattern memorization rather than genuine rule understanding? If so, beyond discovering this phenomenon, does the Active ARC environment contribute in any positive direction to improving reasoning?
  • Is there any rule perturbation test, or any experiment or analysis in which the model is asked to explicitly explain the rule it identified? Given the evidence of memorization, was there an experiment that generated alternations of the original ARC problems (similar but different rules) and compared the results?
  • Can simply passing demonstration example pairs be defined as genuine “teaching”? A model that reveals the example pairs of a static ARC task sequentially would provide more informative example pairs than Astra, so would it count as a better teaching model than Astra?
  • Is there any way to prevent a strategy of exploiting the rule that submitting an invalid input does not deduct from the query budget, in order to narrow down the hypothesis space? On the other hand, producing a grid that is correct down to the fine details and informative sometimes requires already knowing the task’s rule. In such cases, I think a model should also be able to learn the rule while generating invalid grids. I question whether not deducting query budget for invalid grids is the right setting.
  • Isolating why performance drops when active discovery is used: Active-K performs relatively poorly. Is this because the model fails to select high-information inputs, or simply because it fails to produce valid inputs?
6 comments

We thank the reviewer for the detailed and critical reading. In response, we added further analyses of the existing runs as well as several new experiments, which are all part of the revised manuscript (mainly Sec. 4.2 to Sec. 4.4 and Appendix A.9 to A.19). We first answer the points on active exploration and then those on teaching.

W3, Q6: Which capability is lacking, selecting informative inputs or producing valid ones?

▶\blacktriangleright The models lack the ability to select informative inputs, while producing valid inputs is not the problem. Since Active-K and random-K provide the same number of input-output pairs, the gap between them can only have two sources: the model struggles to produce valid inputs, or its valid inputs carry little information about the rule. The revised Sec. 4.2 separates the two, with two further analyses of the existing runs and two new experiments.

Trials without any invalid query. A query that the verifier cannot process is rejected and does not count, so both settings always end with the same number of valid pairs. To test whether the failed attempts still play a role, we split the existing Active-K trials and only keep those in which every query was valid at the first attempt, and compare them to random-K on the same tasks. The gap is almost unchanged: -10.6 / -12.3 points (official / sampled test items), compared to -12.8 / -13.6 points over all trials (A.9).

Valid inputs without any strategy. If validity were the difficulty, any valid input should be about as useful as a generator sample. In a new experiment, we replace the queries of a model by random grids that have the shape and colors of the first example and that the verifier accepts, i.e., valid inputs that require no understanding of the task. On ARC-AGI-2, these random grids are on average as useful as the queries that the models choose deliberately (+0.5 points in favor of the random grids, 95% interval [-2.8, +3.8], four models), and both are considerably less useful than generator samples (A.11).

Wrong rules that the evidence rules out. A query is informative if its output distinguishes the true rule from other rules that are still plausible. We measure this directly for the existing trials. As competing rules we use the incorrect programs that models submit when they are asked to write down the rule as code, restricted to those that agree with the first example of a trial. We then count how many of them are contradicted by the further evidence of that trial. With the queries that the models choose themselves, 28.6% of the competing rules are ruled out on P-ARC, compared to 62.6% with random samples, and the same holds on all four benchmarks (A.12).

Selecting instead of constructing. Finally, in a second new experiment, we remove the construction step entirely. The model receives a pool of unlabeled candidate inputs, half of them its own queries from an earlier trial on the same task and half of them generator inputs, and selects K of them, seeing each output before the next selection. A model that recognized informative inputs would mostly select the generator inputs. Instead, only 39% of the selections are generator inputs, pooled over two models and two benchmarks (A.13).

We summarize this in Sec. 4.2 as follows: "Taken together, the models are able to infer a rule from informative evidence and to produce valid inputs, but do not seem to identify the inputs on which candidate rules differ, whether they construct or select them."

W3: "LLMs do not actually perform active discovery well in this setting"

▶\blacktriangleright The paper is framed to expose this gap and not to resolve it. To our knowledge, it has not been shown before that models which solve ARC tasks from given examples fail to gather such examples themselves. We consequently see the poor performance of the models in the active setting as the finding of the paper, and not as a problem of the setup. The tasks are solvable in this setting, since the same model solves them when it receives the same number of random generator samples instead of its own queries. Active-K is below random-K in all 20 combinations of model and benchmark, by 12.8 points on average, and also on test inputs that are freshly sampled for each trial (13.6 points).

[1/4]

W1: Observation report without methods to resolve the failure

▶\blacktriangleright As mentioned above, the paper evaluates these capabilities and does not propose a method to improve them. In order to make it more useful for developing such methods, we tested the three remedies that one would try first (Sec. 4.4). None of them closes the gap.

An explicit strategy. We add an instruction to the prompt: before each query, the model has to write down at least two different rules that are consistent with everything it has seen, and then choose an input on which these rules would produce different outputs. The models follow the instruction, but all three tested models remain below random-K on P-ARC (official items: GPT-6 Astra 63 compared to 80%, Gemini 3.8 Flash 45 compared to 55%, Claude Opus 5 43 compared to 62%; A.14).

More queries. We give both settings two, four and eight additional pairs. Active-K stays below random-K at every budget, on average by 11.0 points for GPT-6 Astra (95% interval [-18.0, -4.3]) and by 15.3 points for Gemini 3.8 Flash ([-23.7, -7.0]). Even with eight additional queries, Active-K does not reach what random-K achieves with the regular budget, with a single exception (GPT-6 Astra on the official items, 82 compared to 80%; A.4).

More reasoning. We run both settings at high instead of low reasoning effort for three models. Accuracy increases by 18 to 30 points in both settings, but the gap remains in all three cases (A.7).

Providing candidate inputs, in contrast, does help. If the model selects its inputs from a pool of eight generator inputs instead of constructing them, its accuracy rises to the level of random-K (e.g., from 72 to 83% for GPT-6 Astra on P-ARC, with random-K at 80%). Its selections are however only marginally better than random selections from the same pool (by 2.0 to 4.5 points), i.e., the improvement is due to the candidates and not to the selection (A.13). This suggests that a method should focus on proposing and recognizing informative inputs rather than on the reasoning over the evidence. ActiveARC provides what is needed to develop such a method for every task: a verifier that answers any query, a generator of valid inputs, and fresh test items.

Q2: Small and few queries as a sign of memorization instead of rule understanding?

▶\blacktriangleright Memorization would predict the opposite of what we observe. A model that recognizes a task does not depend on its queries, so Active-K would match random-K, whereas it is 10 to 16 points below it on every benchmark. We also tested this directly. For each of the 16 concept groups of ConceptARC, we use five newly generated task families (80 in total) whose rules follow the same concepts but are part of no public benchmark, so that they cannot have been seen during training. On these tasks, Active-K is 16.0 points below random-K (95% interval [-20.1, -12.0]), and it is below for all five models (Sec. 4.4, A.19). The gap is not smaller on rules that the models cannot have seen. In addition, the number of queries is fixed in Active-K, so that a low number of queries cannot explain the gap.

Q3: Rule perturbation tests, or asking the model to state the rule it identified?

▶\blacktriangleright Both are part of the revised paper. The newly generated task families described above correspond to the alternations that the reviewer suggests, since they follow the same concepts as the original ConceptARC tasks but have different rules. In addition, we perturb the rules themselves. For ARC-AGI-1 tasks with a compact reference program, we remove one operation at a time, which gives near-miss rules that agree with the true rule on the first example. The queries that the models choose rule out 50.8% of these near-miss rules, compared to 69.8% for random samples (A.12). The models also have to state the rule explicitly in two settings. In the program-induction variant, the model submits the rule as a Python program, which we execute on held-out examples (A.6). In addition, the instruction described under W1 requires the model to write down the rules it is considering before each query (A.14).

[2/4]

Q5: Should rejected queries be free?

▶\blacktriangleright We agree that there are several options (charging rejected queries against the budget, or limiting their number). We now explain our choice in Sec. 3.2: "We leave rejected queries free instead of charging them against the budget, because an agent cannot know in advance which inputs an unknown rule applies to, and because this can only favor the active setting."

Free rejections can only help the active setting, since it receives information about which inputs the rule applies to that random-K never gets, so the gap we report is conservative. The models also do not exploit this: on ARC-AGI-1 and ARC-AGI-2, 79 to 90% of all trials contain no rejected query at all, and the trials without any rejection show almost the same gap as all trials (see the first point above and A.9).

W2, W3, Q4: Is passing demonstration pairs "teaching"?

▶\blacktriangleright Teaching through a small set of selected examples is the standard formal notion in machine teaching, where a teacher selects the examples from which a learner has to recover a target concept (Goldman & Kearns, 1995; Zhu et al., 2018). It is also what the author of an ARC task does: conveying a rule through a few demonstrations, without observing the solver. We keep this restriction on purpose. Since the teacher has the same information as a task author (the rule, but no contact with the learner), the official training pairs become a like-for-like human reference for every model teacher. With interactive guidance or corrective feedback, the teacher would have information that the task author never had, and the comparison to the human-authored demonstrations would no longer be meaningful.

The framework does support teaching with feedback, and we added a first experiment as a separate setting (A.18). Here, the teacher proposes candidate demonstrations, observes how a learner answers them, and shows the demonstrations that correct the largest number of the learner's wrong answers first. A different student then solves 68 instead of 57 of 80 ConceptARC teaching runs, compared to the same teacher without feedback.

Q4: Would a model that reveals the official pairs be a better teacher than Astra?

▶\blacktriangleright Revealing the official pairs is our static setting, i.e., the human teacher that every model teacher is compared to. Our teachers never see these pairs. They receive the rule and one generated example, and have to construct all further inputs themselves. We verified on the existing runs that they do not reproduce the official examples: none of their demonstration inputs is identical to an official input of the same task on P-ARC and ConceptARC, and at most 1.6% are on ARC-AGI-1 and ARC-AGI-2 (A.15). Compared to this reference, human task authors remain the better teachers, and GPT-6 Astra is the only model that is close (accuracy of the students on the official items, mean over five students):

Demonstrations byP-ARCConceptARCARC-AGI-1ARC-AGI-2
GPT-6 Astra57.880.487.581.4
Human task authors (official pairs)57.482.987.483.7

The other four models are 8 to 17 points below the human-authored demonstrations on average (Sec. 4.3).

References

Goldman, S. A. and Kearns, M. J. (1995). On the complexity of teaching. Journal of Computer and System Sciences, 50(1):20-31.

Zhu, X., Singla, A., Zilles, S. and Rafferty, A. N. (2018). An overview of machine teaching. arXiv preprint arXiv:1801.05927.

[3/4]

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.