acceptodds
Under review as a conference paper at ICLR 2027

ConceptARC at Scale: Studying Conceptual Understanding in Reasoning Models

Abstract

Large reasoning models (LRMs) perform well on challenging benchmarks, but it remains unclear whether they form generalizable abstractions across conceptually related reasoning tasks. The Abstraction and Reasoning Corpus (ARC) provides a controlled testbed for studying abstraction, requiring solvers to infer abstract transformation rules over two-dimensional grids from a small set of examples. While ARC evaluates rule abstraction within individual tasks, different tasks may share higher-level structure despite requiring different transformationsfor example, several tasks may involve reasoning about objects contained within other objects. Benchmarks such as ConceptARC capture this structure by grouping ARC-style tasks into recurring spatial and semantic concepts, but existing datasets contain only a small number of manually constructed tasks per concept. We introduce ConceptARC at Scale, a generator for creating thousands of new ARC-style tasks targeting a specific concept. Using this generator, we construct and publicly release ConceptARC-100, a dataset of 1,600 new, manually reviewed tasks, with 100 tasks for each of the 16 concepts defined by ConceptARCten times as large as the original ConceptARC benchmark. We evaluate a range of recent LRMs, as well as models trained from scratch on ConceptARC-100, to characterize which concepts they solve reliably and where they struggle. We further analyze the models' internal representations and find clear semantic structure: tasks cluster according to the transformation operations applied to objects, such as recoloring, removal, or translation, even when these transformations are realized through substantially different rules and scenarios across tasks. Together, our results suggest that models form stable abstractions of transformation operations that generalize across diverse tasks.

Primary area:
datasets and benchmarks
Keywords:
Concept Understanding, ConceptARC, ARC-AGI, Large Reasoning Models, Rule Induction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RibcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ConceptARC at Scale, a procedural framework for generating ARC-style tasks by composing layouts, selectors, and transformation actions. Based on this framework, the authors construct ConceptARC-100, containing 1,600 tasks with three support examples and ten test queries per task. The paper evaluates eight open-weight reasoning models as well as VARC and TRM, finding considerable variation across concepts and limited consistency at the task level. It further analyzes hidden states and learned task embeddings, reporting that representations become increasingly separable according to transformation action in later layers.

The dataset is potentially useful because it provides executable task programs, concept metadata, and multiple test queries per task. However, the representation results establish correlation with generator-defined action labels more clearly than they establish stable or transferable abstraction.

Critical strengths
  1. ConceptARC-100 substantially expands the original benchmark while retaining executable programs and explicit concept metadata. Ten test queries per task also provide a more reliable measure of within-task consistency than a single query.
  2. The paper evaluates several open-weight model families and combines behavioral results with hidden-state and learned-embedding analyses. This broader evaluation is more informative than testing a single proprietary model or relying only on aggregate accuracy.
Critical weaknesses
  1. The central abstraction claim is confounded by the procedural generator. Shared action implementations can produce recognizable visual or geometric signatures. Without controls that match these low-level properties, clustering by action does not establish that the model learned an abstract transformation that generalizes across different realizations.
  2. Benchmark validity is not sufficiently established. A single research assistant assessed ambiguity and consistency, and 6.2% of tasks required correction, but no independent post-correction audit or human solve study is reported. The fact that models solve some tasks does not show that the remaining tasks are unambiguous or that their support examples uniquely determine the intended rule.
Questions for authors
  1. Does action separation remain after controlling for changed-cell count, output dimensions, color histogram, object displacement, layout family, and action frequency?
  2. How do the representation results compare with raw-grid features and an untrained model?
  3. Can you report an independent post-correction audit, human solve rates, and inter-rater agreement?
  4. Can models generalize to held-out action implementations or unseen component combinations?
  5. Please clarify the numerical inconsistencies in the introduction, Table 1, and the reported number of VARC embeddings.
4 comments
AuthorsOpenReview ↗

Response to Reviewer Ribc 1/2

We thank the reviewer for the thoughtful and constructive feedback. We appreciate the positive comments regarding our dataset, particularly the observation that “ConceptARC-100 substantially expands the original benchmark while retaining executable programs and explicit concept metadata.” We are also pleased that the reviewer recognizes that our evaluation of various open-weight LRMs and our hidden-state analyses provide “more information than testing a single proprietary model or relying only on aggregate accuracy.”

Below, we would like to address each of the reviewer’s concerns:

  1. “Shared action implementations can produce recognizable visual or geometric signatures. Without controls that match these low-level properties, clustering by action does not establish that the model learned an abstract transformation that generalizes across different realizations. Does action separation remain after controlling for changed-cell count, output dimensions, color histogram, object displacement, layout family, and action frequency?”

    This is an excellent question. To address the reviewer’s concern, we added an analysis investigating whether action separation persists after controlling for layout family, action frequency, and visual grid properties, such as grid dimensions, color histograms, and changed-cell counts, as suggested by the reviewer.

    As now described in Section 4.2 (ll. 431–473), we compute 30 visual grid features for each input–output support example (see Table 5 in the Appendix for a description of each feature) and concatenate the feature vectors of the three support examples into a single 90-dimensional vector fvisf_{\text{vis}}. When computing action separation for a reference task tit_i with action label aia_i, we restrict comparisons to pairs of tasks tjt_j and tkt_k whose visual feature vectors fvis,jf_{\text{vis},j} and fvis,kf_{\text{vis},k} are similar. In all control conditions, tasks tjt_j and tkt_k must belong to the same layout family, and action frequencies are balanced across comparisons. To control for visual similarity, we compute the Euclidean distances between the feature vectors of all eligible task pairs and retain only pairs with distances below the 25th, 50th, or 75th percentile of the resulting distribution, with the 25th percentile representing the strictest matching condition (see Figures 17 and 18 in the Appendix for a qualitative comparison).

    We summarize the results in Figure 4b, which shows Qwen3.5-27B’s action separation probability across all layers under this controlled setup. We find that visual matching substantially reduces action-based separation in early and late layers. However, near its peak in intermediate layers, separation decreases by only 1.8%, from 73.4% to 71.6%, even under the strictest matching condition. Under the same controlled setup, we further find that action-based separation remains high (86.4%) for VARC’s task embedding (ll. 481–482).

  2. “How do the representation results compare with raw-grid features?”

    For comparison, we also compute action separation directly from visual feature vectors rather than the model’s task representations. Specifically, we use 132 different visual features per task capturing geometric, color, spatial, and object properties of each grid (see Table 6 for a description of each feature). Under the same matching conditions described above, this visual-feature baseline achieves an action separation probability of only 59.96% (ll. 489–472). Together, these findings suggest that the models’ observed action-based organization cannot be explained solely by the measured visual properties of the support examples. We thank the reviewer for these valuable suggestions, which have substantially strengthened our work.

  3. “A single research assistant assessed ambiguity and consistency, and 6.2% of tasks required correction, but no independent post-correction audit or human solve study is reported. Can you report an independent post-correction audit, human solve rates, and inter-rater agreement?”

    To strengthen benchmark validity, we conducted a second, independent post-correction audit of all 1,600 generated tasks in ConceptARC-100. In this audit, 1,596 of the 1,600 tasks (99.75%) were judged valid. For tasks initially judged valid and therefore left unchanged after the first audit, the two independent annotators agreed in 98.03% of cases. We will revise the four tasks (0.25%) judged invalid in the second audit. However, given the small proportion of invalid tasks, we do not expect these task replacements to notably affect our analyses. Nevertheless, we thank the reviewer for suggesting this additional independent audit, which will help further improve the quality of our dataset.

AuthorsOpenReview ↗

Response to Reviewer Ribc 2/2

Furthermore, we would like to respond to the reviewer’s question:

  1. “Does action separation remain after controlling for changed-cell count, output dimensions, color histogram, object displacement, layout family, and action frequency?“

    Yes. Please find our detailed response to this question in our response to Concern 1.

  2. “How do the representation results compare with raw-grid features and an untrained model?”

    Please find a description of our comparison with a visual baseline using raw-grid features in our response to Concern 2.

    Regarding the second part of the question, could the reviewer please clarify what they mean by the representations of an untrained model in this context? Thank you!

  3. “Can you report an independent post-correction audit, human solve rates, and inter-rater agreement?”

    Please find a description of the second independent post-correction audit and the inter-rater agreement in our response to Concern 3.

    Regarding human solve rates, we agree with the reviewer that this would be interesting to investigate. However, we hope the reviewer understands that asking human participants to solve these tasks is time-consuming, as it requires both reasoning through the problems and producing the correct output grids. Given the scale of our dataset, with 1,600 tasks and 10 queries per task (i.e., 16,000 test queries in total), we are currently unable to conduct such an extensive human evaluation. Nevertheless, we are planning to conduct a study with human participants on our dataset in future work.

  4. “Can models generalize to held-out action implementations or unseen component combinations?”

    We would like to clarify that the pretrained LRMs we evaluate have not been explicitly trained on the tasks in ConceptARC-100. In this sense, we evaluate these models already on held-out action implementations.

    If the reviewer’s question is instead directed at the models we train on ConceptARC-100 (VARC and TRM), we have not evaluated these models on unseen component combinations. However, Deliège et al. [1] investigate the compositional generalization capabilities of VARC on controlled synthetic datasets. They show that, while VARC can generalize to unseen combinations through interpolation when some compositions are present during training, it struggles to execute novel combinations of simple transformations when no such compositions were encountered during training, unless the model backbone is adapted at test time.

  5. “Please clarify the numerical inconsistencies in the introduction, Table 1, and the reported number of VARC embeddings.”

    We thank the reviewer for pointing out these inconsistencies. These were human errors introduced during the final revisions before submission. We have corrected the inconsistencies and updated the Introduction, Table 1, and the corresponding sections of the main text.

We hope this sufficiently addresses the reviewer’s concerns. If not, we would be glad to provide further clarifications. Once again, we thank the reviewer for the constructive and valuable feedback and for taking the time to review our paper!

[1]: Deliège et al. (2026). Implicit Rule Induction with Test-Time Task Embeddings in ARC-like Tasks.

Thank you for the thoughtful reply. The additional visual controls, the independent audit, and the corrections to the inconsistent numbers are all appreciated. However, I still have some concerns so I keep my score unchanged.

  1. The new results are helpful, but they still come from the same generator, so my concern about generalization remains.
  2. I agree with Reviewer REF1 to some extent. With recent progress on ARC-AGI-2 and ARC-AGI-3, I am also unsure what new insights this work offers about model reasoning beyond existing ARC evaluations.
AuthorsOpenReview ↗

Response to Reviewer Ribc

We thank the reviewer for engaging in the discussion and for acknowledging our additional experiments and revisions to the paper.

Regarding the remaining concerns, we would like to briefly address these as well:

  1. “The new results are helpful, but they still come from the same generator, so my concern about generalization remains.”

    Could the reviewer please clarify which specific concerns remain? In our new analysis (Section 4.2, ll. 431–473, Figure 4b), we control for the features directly suggested by the reviewer: layout family, action frequency, and visual properties such as grid dimensions, color histograms, and changed-cell counts. Importantly, we show that even when controlling for these features, action separation remains high for models such as Qwen3.5-27B and VARC. Furthermore, we compute action separation directly from visual grid features (ll. 469–472), as suggested by the reviewer, and show that this visual-feature baseline achieves substantially lower action separation probability. Together, these findings suggest that the models’ observed action-based organization cannot be explained solely by the measured visual properties of the support examples.

  2. “With recent progress on ARC-AGI-2 and ARC-AGI-3, I am also unsure what new insights this work offers about model reasoning beyond existing ARC evaluations.”

    We agree that recent progress on ARC-AGI-2 and ARC-AGI-3 is highly encouraging. However, we would like to emphasize that our work addresses a complementary research question. Rather than primarily measuring whether models can solve increasingly challenging ARC tasks, we investigate which conceptual regularities models capture across tasks, how consistently they apply inferred rules, and how these regularities are reflected in their internal representations.

    In fact, we would like to quote the reviewer’s own words from their original review: “ConceptARC-100 substantially expands the original benchmark while retaining executable programs and explicit concept metadata. Ten test queries per task also provide a more reliable measure of within-task consistency than a single query.” Furthermore, the analyses combine “behavioral results with hidden-state and learned-embedding analyses. This broader evaluation is more informative than testing a single proprietary model or relying only on aggregate accuracy.”

Once again, we thank the reviewer for the constructive feedback and for taking the time to engage with our work! We believe that the reviewer’s suggestions and requests for clarification have substantially strengthened our work.

Reviewer_ov7VOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The ARC-AGI benchmark is a valuable source of non-text-based reasoning, that has historically caused LLMs difficulties. Its tasks involve inferring an underlying transformation from a few example input–output pairs. Different tasks correspond to different transformations.

Data augmentation is well used for ARC-AGI - particularly, colour permutation and rotations/reflections - to extend from "a few" example pairs to many (highly related) pairs. Generators such as RE-ARC extend this to other applications of the same rule. The objective then is for the model to be able to generalise from these 'training' input–output pairs to the 'test' pair.

ConceptARC takes this even further: transformations are grouped by 'concept' (one of 16), and multiple examples of that 'concept' are given. The original ConceptARC dataset only provides 10 examples per concept. This is rather limiting for meaningful statistics to be done. The current dataset, ConceptARC-at-Scale, extends this to 100, allowing meaningful train/test splits for example.

Some recent LLMs are evaluated on the dataset, and some light-weight analysis given. But, the real value is in the dataset - hence the primary area as "datasets and benchmarks" not "experiments"/similar.

Critical strengths

I always thought that the ConceptARC benchmark was nice, but it was just too small to do anything meaningful with: how can I get any statistical significance at 10 examples, especially if I want to do train/[validation/]test splits?

The paper clearly introduces the benchmark, how it is structure and what checks were made. I can certainly see this benchmark being useful in the future. If I work on ARC-AGI, I'll definitely have it in mind to use.

It's an easy "accept" for me to the "datasets and benchmarks" track - modulo the corrections needed that I highlight below.

Critical weaknesses

The obvious question for all this is

How do frontier models perform?

Concept-ARC is largely saturated by even Qwen3.5-27B, never mind GPT-6.x or Opus/Fable 5.x. This is quite possibly due to data leakage: the Qwen3-32B → Qwen3.5-27B jump from ~25% → ~85% is substantial; yet Qwen3 was released almost 2yrs after the original ConceptARC. A small investigation could've been done - eg, roll back to a couple of older LLMs - rather than just state it.

More generally, whilst the dataset is surely of value - even if it's somewhat saturated - and introduced clearly, I found the analysis somewhat lacking in quality.

I do appreciate that the authors want to give some explanation. Quantitative is fairly easy: here's the accuracy on the evals; see Table 1. But, these qualitative analyses are extremely difficult. That said, claims do have to be backed up by the data, and I just don't feel that data-based justification is given for many of the claims.

Below are detailed comments on §4.

  1. §4.1. Evaluating concept-level performance:
    1. Cross-concept comparison:
      1. This comparison across different concepts is nice to see
      2. It would be better as a figure, complementing Table 1, though:
        1. Text/tables are dense, but great for precise information
        2. But, a reader primarily wants a large-scale understanding
      3. Beyond this, how is a causal effect of the concept?
        1. It seems quite possible to me that your "Order" questions are 'objectively' harder, whatever that means
        2. Eg, perhaps there are just many more aspects to process, and humans under a time constraint would perform poorly too?
        3. The claim that the LLMs perform worse on such concepts is backed up by the evals; any suggestion that the concept causes this is not
    2. Table 1:
      1. No error bars are given, and 2dp are displayed
      2. Looking at Table 3, in which there are error bars, this is crazy
      3. There, the errors can be 10 percentage points, and often 10% relative
      4. Providing 2dp without comment indicates much more precision than can be justified
    3. TRM in Figure 1 and §4.1:
      1. HRM & TRM both report >40% on ARC-AGI-1. This makes the reported values, which average 7%, seem suspect
      2. Both require utilise (and likely require) extensive data augmentation, which doesn't appear to have been used here:
        1. Indeed, the end of §4.1 suggests that VARC gets an 8× increase in examples
        2. HRM/TRM typically use 100s, if not 1000, examples per task - all highly correlated augmentations, though
        3. Indeed, 10,000 epochs for three examples intuitively feels like crazy overfitting, no?
      3. This comparison needs to be either removed or done properly (ie, according to TRM's process) - the current is very much 'straw-man'
      4. I don't know the details of VARC so well, so do not comment on the legitimacy of this
  2. §4.2. Understanding how models represent tasks internally
    1. Extracting task representations:
      1. "Specifically, we use the token position at the end of the third support example... At this point, the model... should have sufficient information to infer the underlying task rule" yet you explicitly state that you "evaluate all models in reasoning/thinking mode"
      2. It seems unfair to the LLM to be looking at its representation before thinking, yet evaluating its output that comes after thinking, no?
    2. Organisation by layers:
      1. Qualitative analyses are extremely difficult, in my opinion
      2. The one here seems somewhat ad hoc:
        1. No mechanism is given to suggest why this is reasonable
        2. How can I distinguish it from noise mining?
        3. Even at best, the signal is pretty minor
      3. Again, I appreciate that that the authors want to say something more than "here's this dataset". But, what they say has to be backed up by data
    3. The figure for VARC is actually far more convincing:
      1. Sure, there's lots of overlap in some concepts, but come clearly separate
      2. The claims are much softer than the by-layer ones - specifically, removing the "this changes as we change layers" axis - and more defensible

To be clear, I am likely far more stringent on this than almost any other ML reviewer. I have this complaint with the vast majority of LLM/adjacent papers. It doesn't stop it from being a valid (and important) complaint. But, the authors should be judged fairly when compared with other authors - just because the got me who will actually ask these questions, not just go "oh, yeah, that is cool", shouldn't disadvantage them. But, they should still improve the report!

Also, I spotted (only one) typo: the opening " before Count is missing on line 204.

Questions for authors

I did not have any specific questions to raise with the authors. I did not check the appendix, supplementary material or data myself; instead, I gave an LLM the following prompt, to check the validity more thoroughly than I would be able to.

Please perform a really extensive investigation into the evaluation procedure, see if there are any issues? Standard ones include:

  • Mismatches between figures and data
  • Impossible values (eg, denominator is 100, yet numbers not a multiple of 1/100)
  • Incorrectly calculated error bars/summary statistics (eg, variance of wrong random variable or r-squared calculation)
  • Data leakage in the training procedure (eg, label encodes answer)

Please also look at the supplementary material to see if there are any major issues in that. I do not need you to check every single input–output pair.

Write a short and concise report on your findings in Markdown format. Keep the information self-contained, readable to someone who only has the paper and the supplementary data.

I used the following process:

  • A report was created and checked in one chat (GPT-6.1 Sol/high), using the above prompt
  • It then had an adversarial pass (GPT-6 Luna/high), and corrected two things accordingly with my approval
  • It then launched a follow-up pass (GPT-6 Luna/high) scoped to these two items with my approval

This turned up several points to be addressed. I copy the report below (§ConceptARC at Scale...). I have read it myself, but not verified the individual claims.

ConceptARC at Scale: evaluation and supplementary-data checks

Scope: the submitted paper, including appendices, and the supplied dataset and HTML viewers. Checks covered table arithmetic, figure consistency, possible support/test leakage and selected task semantics. Models were not rerun; this is not a semantic certification of every task.

1. Results and evaluation denominators need reconciliation

Figure 3 does not consistently reproduce Table 1. With ten queries per task and identical task and seed weights, query accuracy must equal the arithmetic mean of the ten survival percentages: each correct query contributes once to that sum. Extracting the plotted vector coordinates gives:

ModelAccuracy implied by Figure 3Table 1
Qwen3.5-27B≈64.47%63.69%
Qwen3.5-9B≈34.48%32.94%
TRM≈7.48%7.43%

The other seven models agree within approximately 0.01 percentage points. The discrepancies above exceed extraction/rounding precision. Different evaluated subsets or seed weighting could explain them, but would need documenting.

Four Table 1 cells are incompatible with three complete, equally weighted seeds. Each concept has 100 tasks × ten queries, so three seeds yield percentage increments of 1/30. Qwen3.5-27B’s CompleteShape 77.45% and Qwen3.5-9B’s CleanUp 29.15%, CompleteShape 44.55% and Copy 28.05% are unattainable even after rounding. All four are compatible with two seeds. The caption’s “evaluated seeds” leaves this possible, but the stated three-seed protocol and Figure 3 caption require clarification.

Two smaller discrepancies:

  • The 16 displayed VARC/TRM concept means give Pearson 0.818 and Spearman 0.712, rather than 0.84/0.74 (§4.1.1). Rounding cannot explain this; a different aggregation might.
  • The introduction’s “fewer than 20%” complete-task success conflicts with 23.77% in §4.1 and Figure 3.

Request: provide per-seed counts/predictions, missing-run handling and the aggregation code, then regenerate the affected summaries together.

2. Two support sets leave natural rules indistinguishable

In the dataset archive, inspect SameDifferent24_001.json and SameDifferent29_001.json, both under SameDifferent/. Both contain objects above and below a horizontal divider. “Shape” below means the translation-normalised occupied-cell pattern, ignoring colour. Retaining an object preserves its coordinates and the input grid dimensions; everything else becomes zero.

  • SameDifferent24: retain the upper object iff the shapes differ; otherwise output a blank grid. This matches every supplied output. However, retaining it iff the colours differ also matches all three supports, but fails tests 2, 3, 4, 5, 6 and 10.
  • SameDifferent29: retain the upper object iff the shapes match; otherwise blank. This matches every output. Again, retaining it iff the colours differ matches all supports, but fails tests 1, 3, 5, 6, 7 and 9.

Indices are 1-based; complete output arrays were checked. Each alternative scores 3/3 supports but 4/10 tests, versus 10/10 for the shape rule. Thus a support-compatible interpretation can cause six test errors without faulty rule execution. These are ambiguities, not inconsistent target functions.

Appendix B’s validation displayed supports and tests together. Please add contrastive supports and check this attribute-confounding pattern more broadly. The targeted examples do not establish a dataset-wide defect rate.

3. Representation measurements have narrower implications

Section 4.2 averages eight support-set representations before measuring separation. This establishes organisation of the averages, not stability between individual support sets. Nor does a higher action score necessarily imply more faithful encoding than a lower concept score: with 16 concepts fully crossed with two actions, equal-weight concatenated one-hot encodings recover both perfectly, yet the stated centring/cosine/triplet procedure scores 75% for concepts and 96.875% for actions.

This is a metric counterexample, not an observed confound. Cross-support and component-matched comparisons would strengthen the interpretation. The supplied viewers contain only 2D coordinates and examples, not original-space vectors or layout/selector/action labels, so the reported separation scores cannot be recalculated from them.

4. Exact answer overlap is small; training provenance remains unclear

Across the core dataset, eight test input–output pairs exactly repeat another task’s support pair; none overlaps within its own task. Example: Order5_001 test 10 equals Order33_001 support 2. For pooled VARC/TRM training, direct exploitation could contribute at most 0.05 percentage points of query accuracy—not explain the main results.

Across all eight support instances in the VARC viewer, 80 of 16,000 core test queries match a viewer support pair (79 unique input–output pairs), giving a conditional direct-overlap bound of 0.50 percentage points. Whether that pool trained any reported evaluator is unspecified. Please link the viewer, Table 1 and the 24-support experiment to their actual training sets/checkpoints, and report the expanded run’s optimisation budget. This is not demonstrated exposure in that experiment. Learned task IDs themselves are legitimate support-trained conditioning, not automatic answer leakage; their evaluation concerns new inputs for trained tasks, not unseen-task transfer.

Checks without a detected contradiction

The 1,600 tasks have the stated three supports and ten tests; all grids are structurally valid. No within-task repeated inputs, support/test input overlap, contradictory mappings or exact duplicate complete tasks were found. Table 1/3 summary means are rounding-compatible, and Table 3’s reported means/sample SDs are arithmetically feasible under its denominators. This does not verify their underlying runs. Prediction logs, training manifests, generator/component metadata and original embeddings are needed to resolve the remaining evaluation questions.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ov7V 1/2

We thank the reviewer for their positive and thoughtful feedback. We appreciate that the reviewer shares our view that the original ConceptARC benchmark is highly valuable and interesting but limited by its size. We also appreciate their recognition that our approach to scaling it up to support more meaningful analyses and sensible “train/validation/test splits” is a useful contribution. We are particularly pleased that the reviewer “can certainly see [our] benchmark being useful” and would consider using it in future work on ARC-AGI.

Below, we would like to respond to the reviewer’s concerns:

  1. “Concept-ARC is largely saturated […] This is quite possibly due to data leakage. A small investigation could've been done - eg, roll back to a couple of older LLMs - rather than just state it.”

    We now explicitly compare performance on ConceptARC (Table 4) with performance on our new ConceptARC-100 benchmark (Table 3), reporting that mean accuracy across the evaluated LRMs is substantially higher on ConceptARC than on ConceptARC-100: 54.39% versus 31.18%, respectively (ll. 304–306). As noted in Section 4.1, training-data contamination may contribute to this difference, although verifying whether the models encountered benchmark items during training is difficult without access to their training data.

    Prior work has evaluated both earlier language models and more recent reasoning models on ConceptARC. For example, Moskvichev et al. [1] reported overall GPT-4 accuracies of approximately 19% and 25% under different sampling settings. Palmarini and Mitchell [2] subsequently evaluated GPT-4V using image-only, text-only, and mixed inputs. As discussed in Section 2 (ll. 182–186), Beger et al. [3] evaluated more recent reasoning models, reporting accuracies of 60.2% for Claude Sonnet 4, 66.0% for Gemini 2.5 Pro, and 77.1% for OpenAI’s o3. Taneja and Banerjee [4] further evaluated DeepSeek-V3.2 as a solver and used Claude Haiku 4.5 to assess conceptual understanding from the solver’s reasoning traces, independently of output correctness.

    These studies provide useful context, including results for older models. However, comparisons across model generations cannot, by themselves, isolate the contribution of training-data contamination. Models may also differ in architecture, training data, and post-training procedures, many details of which are not publicly available. We therefore treat contamination as a possible explanation for the observed performance difference rather than an established cause.

  2. “[The] comparison across different concepts is nice to see […] Beyond this, how is a causal effect of the concept? It seems quite possible to me that your ‘Order’ questions are 'objectively' harder, whatever that means.”

    We thank the reviewer for raising this concern and agree that a measure of task difficulty within each concept category, beyond the reported model accuracies, would be a valuable addition to our paper. We believe that comparing human performance across concepts could provide a complementary indication of task difficulty, particularly if human solvers can reliably recognize the underlying concepts. Our annotation results provide some support for this premise: we asked an independent annotator to assign one of the 16 concept labels to each generated task in ConceptARC-100 and found that the intended concept was identified in 96.75% of cases (ll. 252–256; see Figure 13 in the Appendix for the confusion matrix). These results suggest that the intended concepts are generally recognizable from the tasks’ grid examples. A human-solving evaluation could therefore help assess how difficult the specific tasks are to solve once their underlying concepts have been recognized, rather than merely whether those concepts can be identified.

    We plan to assess human solving performance on our benchmark in a follow-up study. However, asking human participants to solve these tasks is time-consuming, as it requires both reasoning through the problems and producing the correct output grids. Given the scale of our dataset, with 1,600 tasks and 10 queries per task (i.e., 16,000 test queries in total), we are currently unable to conduct such an extensive human evaluation within the scope of the rebuttal period. Nevertheless, we plan such a study for future work and thank the reviewer for the suggestion!

AuthorsOpenReview ↗

Response to Reviewer ov7V 2/2

  1. “Table 1: No error bars are given, and 2dp are displayed.”

    We thank the reviewer for pointing this out. We have added Table 3 to the Appendix, which reports the models’ mean accuracies on ConceptARC-100, together with their standard deviations across evaluation seeds. We would like to highlight that the standard deviations for ConceptARC-100 are substantially lower than those for ConceptARC (Table 4 in the Appendix). This may partly reflect the substantially larger number of tasks and queries per concept in ConceptARC-100: 100 tasks with 10 queries per task, compared with only 10 tasks with 3 queries per task in ConceptARC. We believe that this greater stability of performance estimates across evaluation seeds highlights an important advantage of our new benchmark.

  2. “HRM & TRM both report >40% on ARC-AGI-1. This makes the reported values, which average 7%, seem suspect. Both require utilise (and likely require) extensive data augmentation, which doesn't appear to have been used here.”

    As the reviewer correctly notes, the comparatively poor performance of VARC and TRM may be partly attributable to the fact that we do not “use test-time training, task augmentation, or multiple test-time predictions” (ll. 368–369) when evaluating these models on ConceptARC-100. As further explained in lines 369–371, this is a deliberate choice, “as our goal is to assess how well these models learn the new ConceptARC-100 tasks from their support examples, which concepts are most challenging, and how task information is represented in their embeddings.” We further show that, when the support set is increased eightfold to 24 examples per task, VARC achieves a query accuracy of 77.4%, exceeding the reported accuracies of all evaluated LRMs. This suggests that the amount of task-specific training data is an important performance bottleneck, as highlighted in lines 390–392.

  3. “Organisation by layers: Qualitative analyses are extremely difficult, in my opinion. No mechanism is given to suggest why this is reasonable.”

    We tried our best to describe the separation probability analysis clearly and explain its rationale in Section 4.2 (ll. 395–429). Following Reviewer Ribc’s suggestion, we have also added two control experiments to test whether the observed high action separation probability could be explained by similarities in visual grid features that tasks with the same action label tend to share. Details of these experiments can be found in Section 4.2 (ll. 431–473) and Figure 4b, as well as in our response to Reviewer Ribc’s concerns 1 and 2. Beyond these controls, are there particular aspects of the separation analysis that the reviewer would like us to clarify or address? We would be grateful for any further suggestions.

Furthermore, we would like to thank the reviewer for pointing out the typo in line 204 (which we have now corrected!) and for providing the detailed LLM-generated review of our appendix and supplementary material. We have worked through the LLM response and corrected inconsistencies where appropriate. Once again, we thank the reviewer for their constructive and valuable feedback and for taking the time to review our paper. We hope our response sufficiently addresses the reviewer’s concerns. If not, we would be glad to provide further clarifications!

[1]: Moskvichev et al. (2023). The conceptarc benchmark: Evaluating understanding and generalization in the arc domain.

[2]: Palmarini and Mitchell (2024). Abstract Understanding of Core-Knowledge Concepts: Humans vs. LLMs.

[3]: Beger et al. (2025). Do AI Models Perform Human-like Abstract Reasoning Across Modalities?

[4]: Taneja and Banerjee (2026). Correctness Without Concepts? Evaluating LLM Reasoning Beyond Anthropocentric Taxonomies.

Thank you for your comments. They're all very reasonable. Naturally, I'll keep my score the same - it can't be raised!

Just a few minor follow-up points.

  1. On data leakage, I don't know if it would work here, since the tasks are difficult, but one way to assess data leakage of input–output pairs is to give only the test input and ask for the output:

    • There is no way that a model which hasn't seen the data before would be able to solve this
    • So, if it solves it, even some proportion of time significantly above random, then it's likely that it's seen it before
  2. On human evaluations, I agree with the other reviewer(s) that this would be very valuable. To be honest, even then, humans may also find certain ones harder than others - or, they may find them all so easy that it's hard to get a statistical signal. Still, I agree that it would be beneficial

  3. On precision, that sounds good

  4. On HRM & TRM, the test-time training is really an essential part. There was a workshop paper at NeurIPS last year that looked into what they call "overfitting": instead of doing test-time training across all tasks simultaneously, train on just a single task's input–output pairs. They don't get as good performance as training across everything simultaneously - something something cross-task learning - but they get well above 10%. You could consider a similar mechanism. And, it would be much better in your scenario:

    • increasing 3→24 example pairs would help learning
    • having 10 test pairs reduces noise
  5. On qualitative results, I don't have any particular aspects to highlight - partially, I simply don't remember, and haven't re-read the paper. All I would say is to ensure that any claim that you make is backed up by the data. Ensure that you don't phrase "if" results as "if and only if" - in other words, providing a mechanism for something doesn't mean that it is the only mechanism

Thanks for the good work. Regardless of other reviewers' or ACs' decisions, I'm sure that this will provide value to the community

Reviewer_REF1OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper introduces ConceptARC at scale, a generator for ARC-AGI style reasoning tasks grouped by 16 specific concepts as defined by the ConceptARC dataset. Using the generators, the paper creates the ConceptARC-100 dataset which includes 100 instances per concept. The paper then evaluates open-weight LLMs/LRMs as well as two specialized models on the new dataset. The hidden states of the different models while solving tasks are used to analyze how these models represent different reasoning tasks internally.

Critical strengths

The paper is well-written and easy to read.

The clustering experiments showing that tasks with similar action operations are interesting.

Critical weaknesses

The relationship between predicate concepts, action concepts, generator selectors and generator actions are not clear.

  • Firstly, predicate concepts and action concepts do not seem to be mutually exclusive, which means that the same task can be generated for two different concepts. This can be a problem for the later analysis of how different models represent tasks from different concepts.
  • Secondly, there seems to be an unaddressed connection between generator selectors vs predicate concepts and generator actions vs action concepts. Figure 5 includes a generator action called “extension of object-boundary cells” which seems related to the “ExtendToBoundary” concept. Similarly, Figure 1 has selector named “select_objects_inside” which seems related to the InsideOutside concept.

The evaluation and analysis of existing models is limited. While I share the authors’ concern regarding reliance on proprietary models, the resulting experiments and their corresponding conclusions drawn in section 4.1 can only be applied to open weight models, which limits the usefulness of the study.

The analysis of how LRMs internally represent different tasks draws stronger conclusions that the experiments support. The paper states that LRMs organize tasks by layout early, but Figure 4 shows only Qwen3.5-27B, and that all four categories lie within 55%-65% separation probability. Also worth noting is that the separation probability by layouts decreases initially, but increases again at the end to similar levels as the beginning, this is not discussed.

The generator itself also has limited novelty. Breaking ARC-AGI tasks into a selection and an action step has been a common practice in existing works, which the paper does not discuss. Some examples from the literature includes:

  • [1] Xu et al. "Graphs, constraints, and search for the abstraction and reasoning corpus." 2023
  • [2] Barke et al. "Hysynth: Context-free llm approximation for guiding program synthesis." 2024
  • [3] Batorski et al. "Nsa: Neuro-symbolic arc challenge." 2025
Questions for authors

How are the functions for the generator selectors and actions defined?

Can different concepts result in the same exact task?

Can you clarify on the relationship between the predicate concepts and selectors, as well as action concepts and actions?

4 comments
AuthorsOpenReview ↗

Response to Reviewer REF1 1/2

We thank the reviewer for their valuable and constructive feedback and are grateful to hear that the paper is “well-written and easy to read.” We further appreciate that the reviewer finds our “clustering experiments […] interesting,” and we are encouraged that the reviewer indicated having learned something interesting or useful from reading our paper.

Below, we would like to address each of the reviewer’s concerns:

  1. “The relationship between predicate concepts, action concepts, generator selectors and generator actions are not clear […] There seems to be an unaddressed connection between generator selectors vs. predicate concepts and generator actions vs. action concepts.”

    We appreciate the reviewer’s feedback and acknowledge that our paper would benefit from a clarification of the relationship between predicate concepts and generator selectors, as well as between action concepts and generator actions. We therefore added Figures 14–16 to the Appendix of our paper, illustrating the distributions of layout, selector, and action families across tasks within each concept, with descriptions of each family provided in Tables 8–10. We further clarify in Section 3.1 (ll. 238–244) that concepts describe the semantic properties that tasks are intended to capture, whereas generator components, such as selectors and actions, specify how transformations are implemented. Predicate concepts primarily constrain object selection, while action concepts primarily constrain the transformation. However, there is no one-to-one correspondence: the same component can be used across different concepts, and a concept can be realized through different combinations of components. We hope these additions sufficiently clarify the relationship.

  2. “Predicate concepts and action concepts do not seem to be mutually exclusive, which means that the same task can be generated for two different concepts. […] Can different concepts result in the same exact task?”

    It is true that the concepts originally defined by ConceptARC [1] are not necessarily mutually exclusive, and we now discuss this in Section 3.1 (ll. 248–257). For instance, a task that counts objects inside a container may involve both “Inside and Outside” and “Count.” Such overlap is expected, as a task’s assigned concept describes the primary abstraction it is designed to test, rather than all semantic properties involved in its solution. Notably, we asked an independent annotator to assign one of the 16 concept labels to each generated task in ConceptARC-100 and found that the intended concept was identified in 96.75% of cases (ll. 252—256). We present a confusion matrix of annotation disagreements in Figure 13 of our paper.

    Still, we agree with the reviewer that, for downstream analyses, different concepts should not contain identical tasks. We therefore first verify that no two tasks share an identical support set. Since different support sets may still represent the same transformation rule, we additionally compare task programs across concepts by applying each program to the other task’s input grids. Two tasks are considered duplicates if both programs reproduce each other’s outputs on all tested inputs (Appendix B, ll. 781—792). Importantly, any disagreement on a single input establishes that the two programs are not functionally equivalent. This procedure identifies five pairs of duplicate tasks across concepts (10 out of 1,600 tasks, or 0.625%), which we subsequently replace. Given this small fraction, we do not expect these duplicates to notably affect our analyses in Section 4.2. Nevertheless, we thank the reviewer for highlighting this issue, as addressing it improves the quality of our dataset.

  3. “The resulting experiments and their corresponding conclusions drawn in section 4.1 can only be applied to open weight models.”

    We would like to clarify that while we deliberately chose to promote reproducibility and evaluate only open-weight models (ll. 266–268), our analyses in Section 4.1, such as the concept-specific performance breakdown in Table 1 and the consistency evaluation in Figure 3, are not restricted to open-weight models and can also be applied to proprietary API-based models.

AuthorsOpenReview ↗

Response to Reviewer REF1 2/2

  1. “The paper states that LRMs organize tasks by layout early, but Figure 4 shows only Qwen3.5-27B, and that all four categories lie within 55%-65% separation probability. Also worth noting is that the separation probability by layouts decreases initially, but increases again at the end to similar levels as the beginning, this is not discussed.”

    We agree with the reviewer that presenting results for additional models strengthens the generalizability of our findings. While our initial submission already included results for Gemma4-31B (Figure 6 in the Appendix), we have now added results for GPT-OSS-120B and Qwen3.5-9B (Figure 7 in the Appendix) and refer to them in Section 4.2 (ll. 428–429). We also now discuss the increase in layout separation probability in later layers (ll. 424–426), as suggested by the reviewer. Finally, we would like to clarify that the separation probabilities for the four label categories do not all lie between 55% and 65%: for Qwen3.5-27B, the action separation probability peaks at 73.4% in the intermediate layers (ll. 420–424).

  2. “Breaking ARC-AGI tasks into a selection and an action step has been a common practice in existing works, which the paper does not discuss.”

    We thank the reviewer for drawing our attention to these studies. They are indeed directly related to our work, and we have added a discussion of program synthesis approaches for ARC, including ARGA and the suggested studies [2–4], to Section 2 (ll. 139–150). Similar to our task generator, these approaches express ARC transformation rules through compositions of object-selection and transformation functions. However, while they primarily aim to infer programs that solve existing tasks, we compose layouts, selectors, and actions to generate new ARC-style tasks, using concept-specific constraints to ensure that each task instantiates a targeted spatial or semantic concept. We appreciate the reviewer’s suggestion, which has helped us better contextualize our work.

We also thank the reviewer for their three questions. Questions 1 and 3, concerning the definitions of generator selectors and actions and their relationship to concepts, are addressed in our response to concern 1 above. Question 2, regarding whether different concepts can produce identical tasks, is answered in our response to concern 2.

We hope this sufficiently addresses the reviewer’s concerns. If not, we would be glad to provide further clarifications. Once again, we thank the reviewer for the constructive and valuable feedback and for taking the time to review our paper!

[1]: Moskvichev et al. (2023). The conceptarc benchmark: Evaluating understanding and generalization in the arc domain.

[2]: Xu et al. (2023). Graphs, constraints, and search for the abstraction and reasoning corpus.

[3]: Barke et al. (2024). Hysynth: Context-free llm approximation for guiding program synthesis.

[4]: Batorski et al. (2025). Nsa: Neuro-symbolic arc challenge.

Thank you to the authors for their detailed response and revisions to the paper. I appreciate the effort and have increased my score from 1 to 2. However, some concerns remain.

  • I understand the motivation for focusing on open models. However, it remains unclear whether the observations about the limitations of LRMs would generalize to proprietary models. Given the substantial progress of recent proprietary models on ARC-AGI-2 and ARC-AGI-3, I believe their exclusion is a limitation.

  • I appreciate the additional discussion of related work and the clarification of the differences from existing approaches. While I believe the generator and the dataset will be useful to the community, I am not convinced that scaling up ConceptARC and the empirical findings provide sufficient novelty and significance for ICLR.

AuthorsOpenReview ↗

Response to Reviewer REF1

We thank the reviewer for engaging in the discussion and highly appreciate that they acknowledge our responses and revisions to the paper.

Regarding the remaining concerns, we would like to briefly address these as well:

  1. “I understand the motivation for focusing on open models. However, it remains unclear whether the observations about the limitations of LRMs would generalize to proprietary models.”

    This touches upon a broader discussion about whether and how we should evaluate proprietary models, which may have restricted access, be discontinued at any time, and have underlying mechanisms and architectures that are not publicly disclosed. As discussed in Section 4 (lines 266–269), we deliberately focus on open-weight models to prioritize reproducibility, transparency, and depth of evaluation. This choice is also consistent with ICLR’s emphasis on reproducibility. While we acknowledge the substantial progress made by recent proprietary models, we would also like to highlight that significant progress has been made in open-weight models. Our evaluations and analyses demonstrate that these models are highly capable and exhibit interesting behavioral patterns that warrant investigation in their own right. We therefore stand by our decision to prioritize open science and hope that the reviewer can understand our rationale.

  2. “I appreciate the additional discussion of related work and the clarification of the differences from existing approaches. While I believe the generator and the dataset will be useful to the community, I am not convinced that scaling up ConceptARC and the empirical findings provide sufficient novelty and significance for ICLR.”

    We greatly appreciate the reviewer's recognition of the usefulness of our task generator and dataset to the community. However, we respectfully disagree with the assessment that our contributions lack sufficient novelty or significance. We would like to clarify our main contributions, as described in Section 1 (ll. 102–113): (i) We introduce a generator that automatically creates thousands of new ARC-style tasks organized around the 16 concepts defined by ConceptARC. Importantly, each task is represented by a program that enables the generation of many grid examples; (ii) we publicly release ConceptARC-100, a dataset containing 1,600 new, manually reviewed tasks, making it ten times the size of the original ConceptARC benchmark; (iii) we evaluate a range of recent LRMs, as well as models trained from scratch on ConceptARC-100, characterizing their performance across concepts and demonstrating that even the best-performing model struggles to consistently solve all test inputs within a task; and (iv) we analyze the models’ internal task representations and uncover a clear semantic structure: tasks cluster according to the transformation operations applied to objects, even when those operations arise from substantially different rules, scenarios, and concepts. We believe that these contributions go beyond merely scaling up ConceptARC and, taken together, represent a substantive contribution to the community.

Once again, we thank the reviewer for the constructive feedback and for taking the time to engage with our work! We believe that the reviewer’s suggestions and requests for clarification have substantially strengthened our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Official Comment to Reviewers

We would like to thank the reviewers for their constructive and valuable feedback. Based on their suggestions, we have revised our paper by making the following additions and clarifications:

  1. We added a discussion of program synthesis approaches for ARC (Section 2, ll. 139–150). In particular, we discuss prior work that represents task programs as compositions of filters that select objects and transformations that modify them [1–3]. We discuss how these approaches relate to our task generator and where they differ. We thank reviewer REF1 for drawing our attention to these relevant studies.
  2. We clarified the relationship between predicate concepts and generator selectors, as well as between action concepts and generator actions (Section 3.1, ll. 238–244). We also added Figures 14–16 to the Appendix to show the distributions of layout, selector, and action families across tasks within each concept, and included descriptions of each family in Tables 8–10. Suggested by reviewer REF1.
  3. We discussed the potential semantic overlap between ConceptARC's concept definitions (Section 3.1, ll. 248–257), identified a small fraction of overlapping tasks (0.625%) from different concepts, and subsequently replaced them (Appendix B, ll. 780–792). We also asked a human annotator to assign each task in ConceptARC-100 to the most appropriate of the 16 ConceptARC concepts and found that the annotator identified the intended concept in 96.75% of cases (Section 3.1, ll. 252–257, and Figure 13 in the Appendix). Suggested by reviewer REF1.
  4. We extended our separation probability analysis to include results for additional models (Figure 7 in the Appendix) and clarified that layout separation increases again at later layers for Qwen3.5-27B (ll. 424–426). We also added an analysis investigating whether action separation persists after controlling for layout family, action frequency, and visual grid properties, such as grid dimensions, color histograms, and changed-cell counts (Section 4.2, ll. 431–473, and Figure 4b). Furthermore, we compared the model's action separability with a baseline based on visual grid features extracted from a task’s grid examples. Suggested by reviewers REF1, ov7V, and Ribc.
  5. We conducted a second, independent post-correction audit of the 1,600 generated tasks in ConceptARC-100. In this audit, 99.75% of tasks were judged valid. Suggested by reviewer Ribc.
  6. We directly compare model performance on ConceptARC and ConceptARC-100 (Section 4.1, ll. 304–306). We also added Table 3 to the Appendix, reporting the average accuracy on ConceptARC-100 across the three evaluation seeds, together with the standard deviation for each model. Furthermore, we corrected numerical inconsistencies among the Introduction, Section 4.1, Table 1, and Figure 3. Suggested by reviewers ov7V and Ribc.

We thank the reviewers once again for their time and effort in reviewing our paper. We believe that these additions and clarifications have substantially strengthened our work.

[1]: Xu et al. (2023). Graphs, constraints, and search for the abstraction and reasoning corpus.

[2]: Barke et al. (2024). Hysynth: Context-free llm approximation for guiding program synthesis.

[3]: Batorski et al. (2025). Nsa: Neuro-symbolic arc challenge.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.