- Rating
- 3(+1)
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that existing systematic-generalization benchmarks rely on three simplifications (elemental composition, productivity as the metric, action-explicit goals) and that these hide the inductive and abductive parts of the capability. It introduces TranSGrid, a grid-transformation task: given a 6x6 initial and target board, a model must output a sequence of row, column and block operations that produces the target, scored by execution. Seven Transformers (0.96M to 88M parameters) are trained on 30M random sequences of length 1 to 9; the largest reaches about 80% on a test set drawn from the same generator (lengths 1 to 12) but 55% on TranSGrid and 16% on its hard subset, with the gap present inside the training length range. Two variants put a simplification back: moving the actions onto separate parts of the board (for lengths up to 9), or feeding the decoder the first L-1 gold actions. Both bring solve rates back to roughly the test level, the decoupled variant under both decodings and the gold-prefix variant only under Top@8 scoring, where any of eight beam candidates may count (under greedy decoding it stays 14 points short for the largest model), and the paper concludes that existing tasks miss the inductive and abductive demands.
Critical strengths
The testbed is well built: a small, fully specified action space, instances with a valid solution by construction, and execution-based scoring that accepts any correct sequence. The two controlled variants are a good design, and the gap that survives inside the training length range is a real result.
Critical weaknesses
-
The claim is about existing benchmarks, but nothing is run on SCAN, COGS, PCFG or gSCAN; the link from the two variants to those tasks is an analogy (Figures 3b, 4b). Nor does the paper engage the work that has already analysed these benchmarks: CFQ (Keysers et al., ICLR 2020, arxiv.org/abs/1912.09713), Patel et al. (ACL 2022, arxiv.org/abs/2203.07402) and Csordas et al. (EMNLP 2021, arxiv.org/abs/2108.12284) on SCAN, Qiu et al. (EMNLP 2021, arxiv.org/abs/2109.12243) on gSCAN, SLOG (EMNLP 2023, arxiv.org/abs/2310.15040) on COGS, and the Hupkes et al. taxonomy (NMI 2023, arxiv.org/abs/2210.03050). Hupkes et al. (2020), which the paper cites, already tests systematicity apart from productivity, so "productivity is the only widely shared criterion" is not right.
-
The closest testbeds are not compared against. Compositional-ARC (Mondorf et al., ICLR 2026) composes grid transformations and gets one parenthetical. Program synthesis from input-output examples is the goal-to-program direction the paper calls abductive, with compositional splits already defined (ExeDec, ICLR 2024, arxiv.org/abs/2307.13883), and its DSLs have the precise action space that Section 3.1 says code generation lacks. The recent reasoning-model work on goal-conditioned puzzles with controllable complexity, ARC-AGI-2 (arxiv.org/abs/2505.11831), Shojaee et al. (NeurIPS 2025, arxiv.org/abs/2506.06941) and PlanBench (NeurIPS 2023 D&B, arxiv.org/abs/2206.10498), is absent too, and no LLM is run on TranSGrid. Whether the diagnosis holds beyond 88M-parameter models trained from scratch is open.
-
The reasoning labels add little. Induction rules are blocks whose effects cancel or overwrite, so more rules means more concealment, and both grow with length: three rules need L of at least 9, so 75.5% of the K = 3 instances and all of abductive level 5 lie outside the training range (Figures 3a, 4a). The curves are never separated from each other or from L. The gold-prefix variant leaves a one-step choice among 86 actions, and "not due to shorter output" has no control; under greedy decoding it stays 13 to 18 points below Test for the five largest models (Table 3). A gap between a test set from the training generator and a constructed shift is expected; the question is which shift, and the labels do not answer it.
Questions for authors
-
Can the same seven architectures be trained and evaluated on one existing benchmark, gSCAN or COGS, to show that it fails to separate what TranSGrid separates? If not, the claims should be scoped to the testbed.
-
At fixed concealment level, does solve rate still fall with K? A joint table of K against the abductive level would show whether the two loads are separable.
-
How does TranSGrid relate to Compositional-ARC and to the ExeDec splits, in task definition and in results, and how do current LLMs or reasoning models do on it?
7 comments
Response to Reviewer TRJ4 (2/4)
W3: The reasoning labels add little. ... The curves are never separated from each other or from L. The gold-prefix variant leaves a one-step choice among 86 actions, and "not due to shorter output" has no control ... the question is which shift, and the labels do not answer it.
Thank you for your constructive feedback. We will address your points with two analyses, one holding length fixed and one separating the two reasoning loads with matched controls, and then clarify the gold-prefix variant of TranSGrid (SCAN).
Separating each load from . The table below reports solve rates (%) of ID 7 (greedy) across inductive loads and abductive loads at each fixed reference length (lengths 10–12 are omitted as they additionally involve productivity, and lengths 1–2 as they contain only and ).
| K=0 | K=1 | K=2 | K=3 | A=1 | A=2 | A=3 | A=4 | |
|---|---|---|---|---|---|---|---|---|
| 3 | 100.00 | 97.31 | – | – | 97.75 | – | – | – |
| 4 | 100.00 | 83.28 | – | – | 89.67 | 81.82 | – | – |
| 5 | 100.00 | 76.29 | – | – | 84.24 | 76.65 | – | – |
| 6 | 94.44 | 77.42 | 43.24 | – | 90.70 | 43.23 | 61.70 | – |
| 7 | 93.75 | 73.08 | 39.39 | – | 91.30 | 44.44 | 40.13 | – |
| 8 | 90.00 | 58.82 | 36.31 | – | 66.04 | 42.62 | 35.20 | 39.75 |
| 9 | 91.43 | 50.00 | 20.93 | 8.50 | 75.51 | 31.25 | 28.57 | 10.59 |
At each fixed , solve rates decrease monotonically as increases, and decrease as the abductive level increases in all but two cells (bold). Both loads therefore could lower performance even when length is fixed.
We have added these results to Sec. 4.4.
Separating the two loads with matched controls: If we understand correctly, by "which shift" you mean which property of TranSGrid causes the gap from Test; please let us know if we have misunderstood your point. To test whether each label captures a distinct demand, we built two matched controls from the original instances. Each reduces one demand while keeping the other factors fixed, and both modify only lengths 1–9, since lengths 10–12 additionally involve productivity.
- TranSGrid (Inductive--): We keep the initial board and replace only the induction-rule spans with random actions, so that no induction rule remains. Each instance keeps an identical length, abductive score, and number of changed cells.
- TranSGrid (Abductive--): We keep the initial board and the action types, order, and induction rules, and change only the positions of actions, moving each unit (an induction rule or a standalone action) as a whole to minimize the spatial overlap between units. This lowers the abductive score in 2,086 of 3,600 instances.
Solve rate (%), Greedy / Top@8; Δ is relative to TranSGrid.
| ID | Test | TranSGrid | Inductive-- | Δ | Abductive-- | Δ |
|---|---|---|---|---|---|---|
| 1 | 25.81 / 36.38 | 16.10 / 20.25 | 21.19 / 32.42 | +5.09 / +12.17 | 16.67 / 20.90 | +0.57 / +0.65 |
| 2 | 35.79 / 51.54 | 18.10 / 23.77 | 27.06 / 43.15 | +8.96 / +19.38 | 18.38 / 23.90 | +0.28 / +0.13 |
| 3 | 61.85 / 64.12 | 26.83 / 33.02 | 50.02 / 55.63 | +23.19 / +22.61 | 26.94 / 32.81 | +0.11 / −0.21 |
| 4 | 71.27 / 72.50 | 32.06 / 40.85 | 57.69 / 63.02 | +25.63 / +22.17 | 31.98 / 39.77 | −0.08 / −1.08 |
| 5 | 73.90 / 75.15 | 37.73 / 47.90 | 61.54 / 66.79 | +23.81 / +18.89 | 38.40 / 49.00 | +0.67 / +1.10 |
| 6 | 78.33 / 78.90 | 50.40 / 58.56 | 68.58 / 72.44 | +18.18 / +13.88 | 57.25 / 63.21 | +6.85 / +4.65 |
| 7 | 79.63 / 80.17 | 55.31 / 61.60 | 70.85 / 74.44 | +15.54 / +12.84 | 61.75 / 66.31 | +6.44 / +4.71 |
- Each factor matters on its own. Reducing either induction or abduction while holding the other fixed improves performance.
- Removing induction yields the largest gains (up to +25.63), but no model reaches its Test performance.
- Reducing abduction mainly helps larger models. IDs 6–7 gain +6.44–6.85 (greedy) and +4.65–4.71 (Top@8), while IDs 1–5 change by at most ±1.10. Further analysis shows that, since Abductive-- keeps every induction rule intact, smaller models remain bottlenecked by induction, whereas larger models solve many previously failed instances once abduction is reduced.
TranSGrid (SCAN). This variant was initially designed to reproduce the action-explicit goals of benchmarks such as SCAN. It provides the first gold actions, but the task does not reduce to predicting a single action: the model still needs to execute all given actions to track the board state before it can predict the final one. However, we acknowledge that this setting also introduces factors beyond the abductive demand.
In the revised paper, we have replaced the TranSGrid (Decoupled) and TranSGrid (SCAN) experiments with these two matched controls (Sec. 5.1–5.2).
Response to Reviewer TRJ4 (3/4)
Q1: Can the same seven architectures be trained and evaluated on one existing benchmark, gSCAN or COGS, to show that it fails to separate what TranSGrid separates? If not, the claims should be scoped to the testbed.
Thank you for this suggestion. We trained the same seven models on the original COGS training set and evaluated them on its test and generalization sets. Accuracy (%) is listed below.
| ID | COGS Test | COGS Gen | Gen (Lexical) | Gen (Structural) | TranSGrid Test | TranSGrid |
|---|---|---|---|---|---|---|
| 1 | 99.50 | 63.07 | 73.41 | 1.07 | 25.81 | 16.10 |
| 2 | 99.80 | 66.20 | 77.02 | 1.27 | 35.79 | 18.10 |
| 3 | 99.87 | 65.41 | 76.18 | 0.83 | 61.85 | 26.83 |
| 4 | 99.87 | 58.13 | 67.77 | 0.33 | 71.27 | 32.06 |
| 5 | 99.87 | 60.06 | 70.00 | 0.40 | 73.90 | 37.73 |
| 6 | 99.87 | 48.81 | 56.94 | 0.07 | 78.33 | 50.40 |
| 7 | 99.87 | 60.33 | 70.36 | 0.20 | 79.63 | 55.31 |
COGS does not separate the seven models, or their reasoning demands, in the way TranSGrid does. All seven models are at ceiling on the COGS test set (99.50–99.87) and near zero on its structural cases (0.07–1.27). Lexical accuracy does not increase with model size. In contrast, TranSGrid orders all seven models monotonically on both Test and TranSGrid. Moreover, COGS divides its generalization set by generalization type (lexical vs. structural), whereas TranSGrid quantifies each reasoning load at fine-grained levels (–, –, and levels 1–5) and can vary each one while keeping the other factors fixed.
We have added these results to Appendix E.
Q2: At fixed concealment level, does solve rate still fall with K? A joint table of K against the abductive level would show whether the two loads are separable.
Thank you for the suggestion. The table below reports ID 7 (greedy) solve rates (%) by inductive load and abductive level within the training range (). Numbers of instances are given in parentheses, and "–" indicates cells with fewer than 10 instances.
| A=1 | A=2 | A=3 | A=4 | |
|---|---|---|---|---|
| K=0 | 99.45 (1,095) | 90.91 (22) | – | – |
| K=1 | 87.63 (679) | 77.41 (425) | 54.55 (11) | – |
| K=2 | 10.00 (10) | 38.24 (387) | 40.25 (487) | 38.17 (186) |
| K=3 | – | – | – | 8.50 (294) |
At each abductive level, solve rates decrease as increases. Together with the matched controls in W3, these results suggest that the two loads can be separated.
We hope this addresses your question. We have added this table to Appendix D.
Response to Reviewer TRJ4 (1/4)
Thank you for your comments and constructive suggestions. We address specific weaknesses and questions below.
Thank you for these references and for pointing out where our statements were imprecise.
Experiments on existing benchmarks. Please see our response to Q1, where we train the same seven models on COGS.
Prior analyses of these benchmarks. We have added a discussion of these works to Sec. 2. They study existing benchmarks from the perspectives of split construction (Keysers et al., 2019), training data (Patel et al., 2022), model configuration (Csordás et al., 2021), unsolved and new generalization cases (Qiu et al., 2021; Li et al., 2023), and a taxonomy of generalization research (Hupkes et al., 2023). Our work is complementary: we analyze these benchmarks from the perspective of the reasoning forms they require, and we propose a new testbed, TranSGrid, that avoids the two simplifications in their task format: action-explicit goals and elemental composition.
On productivity. We agree that this sentence is imprecise. Systematicity is the defining property of systematic generalization, so almost every benchmark tests it; we therefore did not list it as a separate criterion. Beyond systematicity, the evaluated criteria are largely task-specific (e.g., substitutivity and localism in PCFG, and verb argument structure alternation in COGS), and productivity is the only one that is both shared across most tasks and challenging.
We have revised the sentence in Sec. 3.1 to: "Beyond these construction challenges, there is also no consensus on how to measure systematic generalization: apart from systematicity, most criteria are task-specific, with productivity being the only one that is both widely shared across tasks and challenging."
Thank you for these references. We compare TranSGrid with Compositional-ARC and ExeDec, and report LLM results, in Q3. The remaining works and results of larger models are discussed below.
Reasoning-model puzzles. Compared with ARC-AGI-2, Tower of Hanoi and related puzzles (Shojaee et al.), and PlanBench, TranSGrid has two advantages for studying systematic generalization.
We therefore see these benchmarks as complementary: they measure overall problem-solving ability, whereas TranSGrid isolates and separately evaluates the inductive and abductive demands that systematic generalization requires. We have added a discussion of these works to Sec. 2.
Beyond 88M parameters. We trained two models larger than ID 7 with the same training setup. And the results are listed below.
Solve rate (%), Greedy / Top@8:
Scaling up beyond ID 7 does not help close the gap between Test and TranSGrid: ID 8 improves only slightly, and ID 9 even performs slightly worse than IDs 7 and 8.
We have added these results to Appendix G.