- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ActiveARC, an interactive environments for abstract reasoning benchmarks like ARC using generator and verifier programs. This could lead to whether models can choose good examples by themselves, not only infer a rule from given ones. On four ARC-style datasets with five LLMs, self-selected queries are worse than random generator samples.
Critical strengths
-
The setup is simple, straightforward and interesting. Using generator and verifier programs to make ARC interactive is a very interesting idea, and comparing Active-K with Random-K at exactly the same number of pairs is a clean design.
-
Methods are compared cleanly. Active-K and Random-K get exactly the same number of pair, and Active-K is lower in all 20 cell.
-
There are many robustness checks included in the paper, like abandoned trials, a larger budget, noisy outputs and slippage.
Critical weaknesses
-
Only compare the model with Random-K baseline is not fair. It samples from the hand-written generator, which already encodes the factors of variation of each task, but the model has no such knowledge.
-
The teaching claim relies on saturated benchmarks. Static is 93-99.5% on ARC-AGI-1/2 for the top models, so contamination is a concern.
-
All main runs use low effort, and Gemini answers 36-84% of exam items without reasoning.
Questions for authors
-
I'm wondering if it is possible to add a baseline with random valid grids or with simple perturbations of the initial sample? This would show how much of the Random-K advantage comes from the generator's knowledge.
-
How do the main results change with medium or high reasoning effort for more than one model?
3 comments
W2: Does the teaching claim rely on saturated benchmarks, and is contamination a concern?
ARC-AGI-1 and ARC-AGI-2 are indeed close to saturated for the strongest students, which we note in the limitations. The teaching results do not depend on these cases, however. Each teaching result is a combination of a student and a benchmark, and of these 20 combinations, ten have a static accuracy below 90% (down to 10% for GPT-5.6 Luna on P-ARC). To show this, we split the existing results by the static accuracy of the student (A.16). The table gives the mean difference between a teacher's demonstrations and the human-authored ones (points, official / sampled test items):
| Teacher | All (20) | Static below 90% (10) | Static below 80% (7) | Static 90% or above (10) |
|---|---|---|---|---|
| GPT-6 Astra | -1.1 / -1.2 | -0.2 / -1.9 | +0.4 / -2.2 | -1.9 / -0.6 |
| Claude Opus 5 | -8.3 / -4.8 | -8.4 / -4.3 | -8.7 / -3.7 | -8.2 / -5.4 |
| Gemini 3.8 Flash | -8.9 / -7.2 | -8.0 / -8.1 | -8.0 / -9.2 | -9.8 / -6.2 |
| DeepSeek V4.1 Flash | -9.9 / -8.2 | -9.6 / -8.7 | -9.6 / -10.0 | -10.2 / -7.6 |
| GPT-5.6 Luna | -16.9 / -13.7 | -15.7 / -12.3 | -16.4 / -13.8 | -18.1 / -15.1 |
The results do not change when the saturated cases are excluded. GPT-6 Astra stays within about two points of the human-authored demonstrations in every subset, and all other teachers remain below them. The cases at 90% or above also differentiate between the teachers, with losses of 8 to 18 points for the four weaker ones, i.e., the students depend on the demonstrations there as well.
Regarding contamination, there are four reasons why we do not think that it explains the teaching results. First, the results are the same on the sampled test items, which are freshly drawn from the generator and cannot have been seen before (second number in each cell of the table). Second, the teachers do not reproduce the official examples. Of their demonstration inputs, none is identical to an official input of the same task on P-ARC and ConceptARC, and at most 1.6% are on ARC-AGI-1 and ARC-AGI-2 (A.15). Third, the VARC student of the submitted version is trained from scratch on the demonstrations of each task alone and cannot be contaminated. With this student, the ranking of the teachers closely mirrors the one obtained with LLM students (A.24). Fourth, familiarity with the official tasks would favor the static setting, since these are the demonstrations that a model may have seen during training. It would therefore make the teachers look worse and not better.
W3, Q2: Low reasoning effort, and Gemini answering test items without reasoning
We ran both settings at high reasoning effort for three further models as a new experiment (Sec. 4.4, A.7; one draw per cell). Accuracy on official / sampled test items:
| Benchmark | Model | Low effort Active-K | Low effort Random-K | High effort Active-K | High effort Random-K |
|---|---|---|---|---|---|
| P-ARC | Gemini 3.8 Flash | 50 / 54 | 55 / 69 | 68 / 76 | 76 / 92 |
| P-ARC | Claude Opus 5 | 47 / 55 | 62 / 73 | 72 / 74 | 82 / 94 |
| ARC-AGI-2 | GPT-5.6 Luna | 28 / 39 | 47.2 / 56.5 | 56 / 60 | 75.5 / 82 |
High effort raises the accuracy in both settings by 18 to 30 points, but Active-K remains below random-K in all three cases and on both item types. More reasoning seems to make the models better at inferring a rule from the evidence, but it does not help them to choose the evidence. We could not repeat all 20 combinations of model and benchmark at high effort due to the cost of these runs.
Regarding the teacher's own test items, the reviewer is right that Gemini 3.8 Flash skips the reasoning at low effort. Before teaching, each teacher has to solve the test inputs of the task itself, and at low effort Gemini answers most of them without any reasoning. In a new experiment, we therefore repeat its teaching runs on P-ARC and ARC-AGI-2 at high effort, with the students unchanged (A.17). The teacher then reasons on every test input and solves 97.0% (P-ARC) and 98.3% (ARC-AGI-2) of them, compared to 38.0% and 73.2% at low effort. Its demonstrations improve as well: on P-ARC, three of its four students are 8 to 16 points more accurate than with the low-effort teacher (GPT-6 Astra 86 compared to 74%, Gemini 3.8 Flash 68 compared to 60%, DeepSeek V4.1 Flash 58 compared to 42%), while the changes on ARC-AGI-2 are small. The reasoning effort does matter for the teaching quality of this model. Our observation that being able to apply a rule is not sufficient for teaching it does not depend on Gemini 3.8 Flash, however, since Claude Opus 5 and DeepSeek V4.1 Flash solve 81 to 94% of their own test items and still teach considerably worse than GPT-6 Astra.
We appreciate the reviewer's positive view of the setup and the three precise concerns. Each of them is now covered by an experiment or analysis in the revised manuscript, and we welcome any remaining questions.
[2/2]
Thanks for the author's detailed rebuttal! But I still have some concerns regarding the generalizability of the proposed method to broader datasets. Therefore I would like to keep my score.
We thank the reviewer for the positive assessment of the design and for the three concrete concerns. We addressed each of them with new experiments or further analyses of the existing runs, which are all part of the revised manuscript (mainly Sec. 3.3, Sec. 4.2, Sec. 4.4 and Appendix A.7 to A.17).
W1, Q1: Is random-K a fair baseline, and how do baselines without the generator's knowledge perform?
▶ It is correct that the generator's inputs follow the input distribution of the task, which the model does not know in advance. We still think that random-K is the appropriate reference, for the following reason. The generator provides valid inputs, but these are drawn at random. Random-K is in that sense an uninformed selection, since the generator does not know which rules the model is considering and never sees an output. The model, however, sees the output of each query before it chooses the next one, and can choose the input that separates the rules it is considering. Such an informed selection is expected to be better than a random one, which is why random sampling is the standard reference for active learning. The model, the number of pairs and the test items are identical in both settings. We now state this in Sec. 3.3: "[...] the only difference is whether the agent chose the inputs itself or whether they were sampled from the generator, whose inputs are valid but not selected to be informative, while the agent can adapt each query to earlier outputs."
We also added the two baselines that the reviewer suggests as a new experiment (Sec. 4.2, A.11). Both provide the model with valid inputs that require no knowledge of the task. Perturbations recolor 5 to 30% of the cells of the first example, or swap two of its colors. Random grids are uniformly random grids with the shape and colors of the first example. In both cases, only inputs that the verifier accepts are kept, and their outputs come from the same verifier. The table shows the random grids (accuracy on the official test items):
*Abandoned Active-K trials count as incorrect; without them, random grids still lead.
The queries that the models choose themselves are not consistently better than inputs that are chosen without any knowledge of the task. Active-K is below the random grids in four of the eight cells (DeepSeek V4.1 Flash on both benchmarks, Gemini 3.8 Flash on P-ARC and GPT-5.6 Luna on ARC-AGI-2), and averaged over the four models on ARC-AGI-2, the models' queries are as useful as random valid grids (+0.5 points in favor of the random grids, 95% interval [-2.8, +3.8]). The perturbations give a similar picture (A.11). At the same time, the random grids are valid inputs as well, but remain below random-K in every cell. The advantage of random-K over Active-K can therefore not be explained by the generator knowing which inputs are valid. It rather depends on which inputs are observed, and random-K, as described above, does not select them in any way.
Three further results show that the gap is not an artifact of what the generator knows. First, random samples are far from the most informative evidence. On P-ARC, they rule out only 62.6% of the incorrect rules that are consistent with the first example (A.12), so that a query which targets the remaining ones would be more informative than a random sample. The queries of the models rule out 28.6%. Second, a model that knows the rule can exceed the generator. In the teaching setting, the model is told the rule and selects which inputs to show. The demonstrations of GPT-6 Astra are more useful for the students than random-K on three of four benchmarks (57.8, 80.4 and 87.5% compared to 52.0, 73.3 and 84.7% on P-ARC, ConceptARC and ARC-AGI-1; ARC-AGI-2: 81.4 compared to 82.8%; A.15). Third, providing the generator's inputs is not sufficient. In a new experiment, the model selects its inputs from a pool that contains its own earlier queries and the same number of generator inputs, without any indication of their source. Only 39% of its selections are generator inputs (A.13). Taken together, the gap reflects that the models do not select the inputs that discriminate between candidate rules, which is what the active setting is designed to measure.
[1/2]