- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ConceptARC at Scale, a procedural framework for generating ARC-style tasks by composing layouts, selectors, and transformation actions. Based on this framework, the authors construct ConceptARC-100, containing 1,600 tasks with three support examples and ten test queries per task. The paper evaluates eight open-weight reasoning models as well as VARC and TRM, finding considerable variation across concepts and limited consistency at the task level. It further analyzes hidden states and learned task embeddings, reporting that representations become increasingly separable according to transformation action in later layers.
The dataset is potentially useful because it provides executable task programs, concept metadata, and multiple test queries per task. However, the representation results establish correlation with generator-defined action labels more clearly than they establish stable or transferable abstraction.
Critical strengths
- ConceptARC-100 substantially expands the original benchmark while retaining executable programs and explicit concept metadata. Ten test queries per task also provide a more reliable measure of within-task consistency than a single query.
- The paper evaluates several open-weight model families and combines behavioral results with hidden-state and learned-embedding analyses. This broader evaluation is more informative than testing a single proprietary model or relying only on aggregate accuracy.
Critical weaknesses
- The central abstraction claim is confounded by the procedural generator. Shared action implementations can produce recognizable visual or geometric signatures. Without controls that match these low-level properties, clustering by action does not establish that the model learned an abstract transformation that generalizes across different realizations.
- Benchmark validity is not sufficiently established. A single research assistant assessed ambiguity and consistency, and 6.2% of tasks required correction, but no independent post-correction audit or human solve study is reported. The fact that models solve some tasks does not show that the remaining tasks are unambiguous or that their support examples uniquely determine the intended rule.
Questions for authors
- Does action separation remain after controlling for changed-cell count, output dimensions, color histogram, object displacement, layout family, and action frequency?
- How do the representation results compare with raw-grid features and an untrained model?
- Can you report an independent post-correction audit, human solve rates, and inter-rater agreement?
- Can models generalize to held-out action implementations or unseen component combinations?
- Please clarify the numerical inconsistencies in the introduction, Table 1, and the reported number of VARC embeddings.
4 comments
Response to Reviewer Ribc 2/2
Furthermore, we would like to respond to the reviewer’s question:
-
“Does action separation remain after controlling for changed-cell count, output dimensions, color histogram, object displacement, layout family, and action frequency?“
Yes. Please find our detailed response to this question in our response to Concern 1.
-
“How do the representation results compare with raw-grid features and an untrained model?”
Please find a description of our comparison with a visual baseline using raw-grid features in our response to Concern 2.
Regarding the second part of the question, could the reviewer please clarify what they mean by the representations of an untrained model in this context? Thank you!
-
“Can you report an independent post-correction audit, human solve rates, and inter-rater agreement?”
Please find a description of the second independent post-correction audit and the inter-rater agreement in our response to Concern 3.
Regarding human solve rates, we agree with the reviewer that this would be interesting to investigate. However, we hope the reviewer understands that asking human participants to solve these tasks is time-consuming, as it requires both reasoning through the problems and producing the correct output grids. Given the scale of our dataset, with 1,600 tasks and 10 queries per task (i.e., 16,000 test queries in total), we are currently unable to conduct such an extensive human evaluation. Nevertheless, we are planning to conduct a study with human participants on our dataset in future work.
-
“Can models generalize to held-out action implementations or unseen component combinations?”
We would like to clarify that the pretrained LRMs we evaluate have not been explicitly trained on the tasks in ConceptARC-100. In this sense, we evaluate these models already on held-out action implementations.
If the reviewer’s question is instead directed at the models we train on ConceptARC-100 (VARC and TRM), we have not evaluated these models on unseen component combinations. However, Deliège et al. [1] investigate the compositional generalization capabilities of VARC on controlled synthetic datasets. They show that, while VARC can generalize to unseen combinations through interpolation when some compositions are present during training, it struggles to execute novel combinations of simple transformations when no such compositions were encountered during training, unless the model backbone is adapted at test time.
-
“Please clarify the numerical inconsistencies in the introduction, Table 1, and the reported number of VARC embeddings.”
We thank the reviewer for pointing out these inconsistencies. These were human errors introduced during the final revisions before submission. We have corrected the inconsistencies and updated the Introduction, Table 1, and the corresponding sections of the main text.
We hope this sufficiently addresses the reviewer’s concerns. If not, we would be glad to provide further clarifications. Once again, we thank the reviewer for the constructive and valuable feedback and for taking the time to review our paper!
[1]: Deliège et al. (2026). Implicit Rule Induction with Test-Time Task Embeddings in ARC-like Tasks.
Thank you for the thoughtful reply. The additional visual controls, the independent audit, and the corrections to the inconsistent numbers are all appreciated. However, I still have some concerns so I keep my score unchanged.
- The new results are helpful, but they still come from the same generator, so my concern about generalization remains.
- I agree with Reviewer REF1 to some extent. With recent progress on ARC-AGI-2 and ARC-AGI-3, I am also unsure what new insights this work offers about model reasoning beyond existing ARC evaluations.
Response to Reviewer Ribc
We thank the reviewer for engaging in the discussion and for acknowledging our additional experiments and revisions to the paper.
Regarding the remaining concerns, we would like to briefly address these as well:
-
“The new results are helpful, but they still come from the same generator, so my concern about generalization remains.”
Could the reviewer please clarify which specific concerns remain? In our new analysis (Section 4.2, ll. 431–473, Figure 4b), we control for the features directly suggested by the reviewer: layout family, action frequency, and visual properties such as grid dimensions, color histograms, and changed-cell counts. Importantly, we show that even when controlling for these features, action separation remains high for models such as Qwen3.5-27B and VARC. Furthermore, we compute action separation directly from visual grid features (ll. 469–472), as suggested by the reviewer, and show that this visual-feature baseline achieves substantially lower action separation probability. Together, these findings suggest that the models’ observed action-based organization cannot be explained solely by the measured visual properties of the support examples.
-
“With recent progress on ARC-AGI-2 and ARC-AGI-3, I am also unsure what new insights this work offers about model reasoning beyond existing ARC evaluations.”
We agree that recent progress on ARC-AGI-2 and ARC-AGI-3 is highly encouraging. However, we would like to emphasize that our work addresses a complementary research question. Rather than primarily measuring whether models can solve increasingly challenging ARC tasks, we investigate which conceptual regularities models capture across tasks, how consistently they apply inferred rules, and how these regularities are reflected in their internal representations.
In fact, we would like to quote the reviewer’s own words from their original review: “ConceptARC-100 substantially expands the original benchmark while retaining executable programs and explicit concept metadata. Ten test queries per task also provide a more reliable measure of within-task consistency than a single query.” Furthermore, the analyses combine “behavioral results with hidden-state and learned-embedding analyses. This broader evaluation is more informative than testing a single proprietary model or relying only on aggregate accuracy.”
Once again, we thank the reviewer for the constructive feedback and for taking the time to engage with our work! We believe that the reviewer’s suggestions and requests for clarification have substantially strengthened our work.
Response to Reviewer Ribc 1/2
We thank the reviewer for the thoughtful and constructive feedback. We appreciate the positive comments regarding our dataset, particularly the observation that “ConceptARC-100 substantially expands the original benchmark while retaining executable programs and explicit concept metadata.” We are also pleased that the reviewer recognizes that our evaluation of various open-weight LRMs and our hidden-state analyses provide “more information than testing a single proprietary model or relying only on aggregate accuracy.”
Below, we would like to address each of the reviewer’s concerns:
“Shared action implementations can produce recognizable visual or geometric signatures. Without controls that match these low-level properties, clustering by action does not establish that the model learned an abstract transformation that generalizes across different realizations. Does action separation remain after controlling for changed-cell count, output dimensions, color histogram, object displacement, layout family, and action frequency?”
This is an excellent question. To address the reviewer’s concern, we added an analysis investigating whether action separation persists after controlling for layout family, action frequency, and visual grid properties, such as grid dimensions, color histograms, and changed-cell counts, as suggested by the reviewer.
As now described in Section 4.2 (ll. 431–473), we compute 30 visual grid features for each input–output support example (see Table 5 in the Appendix for a description of each feature) and concatenate the feature vectors of the three support examples into a single 90-dimensional vector fvis. When computing action separation for a reference task ti with action label ai, we restrict comparisons to pairs of tasks tj and tk whose visual feature vectors fvis,j and fvis,k are similar. In all control conditions, tasks tj and tk must belong to the same layout family, and action frequencies are balanced across comparisons. To control for visual similarity, we compute the Euclidean distances between the feature vectors of all eligible task pairs and retain only pairs with distances below the 25th, 50th, or 75th percentile of the resulting distribution, with the 25th percentile representing the strictest matching condition (see Figures 17 and 18 in the Appendix for a qualitative comparison).
We summarize the results in Figure 4b, which shows Qwen3.5-27B’s action separation probability across all layers under this controlled setup. We find that visual matching substantially reduces action-based separation in early and late layers. However, near its peak in intermediate layers, separation decreases by only 1.8%, from 73.4% to 71.6%, even under the strictest matching condition. Under the same controlled setup, we further find that action-based separation remains high (86.4%) for VARC’s task embedding (ll. 481–482).
“How do the representation results compare with raw-grid features?”
For comparison, we also compute action separation directly from visual feature vectors rather than the model’s task representations. Specifically, we use 132 different visual features per task capturing geometric, color, spatial, and object properties of each grid (see Table 6 for a description of each feature). Under the same matching conditions described above, this visual-feature baseline achieves an action separation probability of only 59.96% (ll. 489–472). Together, these findings suggest that the models’ observed action-based organization cannot be explained solely by the measured visual properties of the support examples. We thank the reviewer for these valuable suggestions, which have substantially strengthened our work.
“A single research assistant assessed ambiguity and consistency, and 6.2% of tasks required correction, but no independent post-correction audit or human solve study is reported. Can you report an independent post-correction audit, human solve rates, and inter-rater agreement?”
To strengthen benchmark validity, we conducted a second, independent post-correction audit of all 1,600 generated tasks in ConceptARC-100. In this audit, 1,596 of the 1,600 tasks (99.75%) were judged valid. For tasks initially judged valid and therefore left unchanged after the first audit, the two independent annotators agreed in 98.03% of cases. We will revise the four tasks (0.25%) judged invalid in the second audit. However, given the small proportion of invalid tasks, we do not expect these task replacements to notably affect our analyses. Nevertheless, we thank the reviewer for suggesting this additional independent audit, which will help further improve the quality of our dataset.