acceptodds
Under review as a conference paper at ICLR 2027

What Do Current Systematic Generalization Tasks Miss? A Reasoning-Centered Analysis

Abstract

Systematic generalization, the ability to solve novel problems by recombining known atomic elements, is central to human intelligence but difficult to study rigorously under controlled settings. Existing studies therefore rely on simplifications such as elemental composition, productivity-based tests, and action-explicit goals, which make systematic generalization easier to study but omit some essential aspects of this capability. To characterize what these simplifications miss, we adopt a reasoning-centered lens and introduce TranSGrid, a testbed that brings deductive, inductive, and abductive reasoning together within a unified task. Experiments with seven Transformer models on 4,800 TranSGrid instances show that all models perform much worse on TranSGrid than on a held-out test set: the largest model solves 79.6% of the test set, but only 55.3% of TranSGrid and 15.8% of the hardest subset. The gap remains within the training length range, showing that productivity alone is not sufficient to evaluate systematic generalization. Additionally, we reintroduce the other two simplifications into TranSGrid: one variant limits interactions among action effects to approximate elemental composition (reducing the inductive demand); the other makes goals action-explicit (reducing the abductive one). In both, solve rates return to roughly the test set level, showing that either simplification alone is enough to reduce TranSGrid to an ordinary held-out test set. Together, our results show that existing tasks reduce either or both of the inductive and abductive demands of systematic generalization, and that comprehensively measuring this capability requires a task that involves all three forms of reasoning.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Systematic Generalization, Logical Reasoning, Task Analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TRJ4OpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that existing systematic-generalization benchmarks rely on three simplifications (elemental composition, productivity as the metric, action-explicit goals) and that these hide the inductive and abductive parts of the capability. It introduces TranSGrid, a grid-transformation task: given a 6x6 initial and target board, a model must output a sequence of row, column and block operations that produces the target, scored by execution. Seven Transformers (0.96M to 88M parameters) are trained on 30M random sequences of length 1 to 9; the largest reaches about 80% on a test set drawn from the same generator (lengths 1 to 12) but 55% on TranSGrid and 16% on its hard subset, with the gap present inside the training length range. Two variants put a simplification back: moving the actions onto separate parts of the board (for lengths up to 9), or feeding the decoder the first L-1 gold actions. Both bring solve rates back to roughly the test level, the decoupled variant under both decodings and the gold-prefix variant only under Top@8 scoring, where any of eight beam candidates may count (under greedy decoding it stays 14 points short for the largest model), and the paper concludes that existing tasks miss the inductive and abductive demands.

Critical strengths

The testbed is well built: a small, fully specified action space, instances with a valid solution by construction, and execution-based scoring that accepts any correct sequence. The two controlled variants are a good design, and the gap that survives inside the training length range is a real result.

Critical weaknesses
  1. The claim is about existing benchmarks, but nothing is run on SCAN, COGS, PCFG or gSCAN; the link from the two variants to those tasks is an analogy (Figures 3b, 4b). Nor does the paper engage the work that has already analysed these benchmarks: CFQ (Keysers et al., ICLR 2020, arxiv.org/abs/1912.09713), Patel et al. (ACL 2022, arxiv.org/abs/2203.07402) and Csordas et al. (EMNLP 2021, arxiv.org/abs/2108.12284) on SCAN, Qiu et al. (EMNLP 2021, arxiv.org/abs/2109.12243) on gSCAN, SLOG (EMNLP 2023, arxiv.org/abs/2310.15040) on COGS, and the Hupkes et al. taxonomy (NMI 2023, arxiv.org/abs/2210.03050). Hupkes et al. (2020), which the paper cites, already tests systematicity apart from productivity, so "productivity is the only widely shared criterion" is not right.

  2. The closest testbeds are not compared against. Compositional-ARC (Mondorf et al., ICLR 2026) composes grid transformations and gets one parenthetical. Program synthesis from input-output examples is the goal-to-program direction the paper calls abductive, with compositional splits already defined (ExeDec, ICLR 2024, arxiv.org/abs/2307.13883), and its DSLs have the precise action space that Section 3.1 says code generation lacks. The recent reasoning-model work on goal-conditioned puzzles with controllable complexity, ARC-AGI-2 (arxiv.org/abs/2505.11831), Shojaee et al. (NeurIPS 2025, arxiv.org/abs/2506.06941) and PlanBench (NeurIPS 2023 D&B, arxiv.org/abs/2206.10498), is absent too, and no LLM is run on TranSGrid. Whether the diagnosis holds beyond 88M-parameter models trained from scratch is open.

  3. The reasoning labels add little. Induction rules are blocks whose effects cancel or overwrite, so more rules means more concealment, and both grow with length: three rules need L of at least 9, so 75.5% of the K = 3 instances and all of abductive level 5 lie outside the training range (Figures 3a, 4a). The curves are never separated from each other or from L. The gold-prefix variant leaves a one-step choice among 86 actions, and "not due to shorter output" has no control; under greedy decoding it stays 13 to 18 points below Test for the five largest models (Table 3). A gap between a test set from the training generator and a constructed shift is expected; the question is which shift, and the labels do not answer it.

Questions for authors
  1. Can the same seven architectures be trained and evaluated on one existing benchmark, gSCAN or COGS, to show that it fails to separate what TranSGrid separates? If not, the claims should be scoped to the testbed.

  2. At fixed concealment level, does solve rate still fall with K? A joint table of K against the abductive level would show whether the two loads are separable.

  3. How does TranSGrid relate to Compositional-ARC and to the ExeDec splits, in task definition and in results, and how do current LLMs or reasoning models do on it?

7 comments
AuthorsOpenReview ↗

Response to Reviewer TRJ4 (1/4)

Thank you for your comments and constructive suggestions. We address specific weaknesses and questions below.

W1: The claim is about existing benchmarks, but nothing is run on SCAN, COGS, PCFG or gSCAN ... Nor does the paper engage the work that has already analysed these benchmarks ... "productivity is the only widely shared criterion" is not right.

Thank you for these references and for pointing out where our statements were imprecise.

Experiments on existing benchmarks. Please see our response to Q1, where we train the same seven models on COGS.

Prior analyses of these benchmarks. We have added a discussion of these works to Sec. 2. They study existing benchmarks from the perspectives of split construction (Keysers et al., 2019), training data (Patel et al., 2022), model configuration (Csordás et al., 2021), unsolved and new generalization cases (Qiu et al., 2021; Li et al., 2023), and a taxonomy of generalization research (Hupkes et al., 2023). Our work is complementary: we analyze these benchmarks from the perspective of the reasoning forms they require, and we propose a new testbed, TranSGrid, that avoids the two simplifications in their task format: action-explicit goals and elemental composition.

On productivity. We agree that this sentence is imprecise. Systematicity is the defining property of systematic generalization, so almost every benchmark tests it; we therefore did not list it as a separate criterion. Beyond systematicity, the evaluated criteria are largely task-specific (e.g., substitutivity and localism in PCFG, and verb argument structure alternation in COGS), and productivity is the only one that is both shared across most tasks and challenging.

We have revised the sentence in Sec. 3.1 to: "Beyond these construction challenges, there is also no consensus on how to measure systematic generalization: apart from systematicity, most criteria are task-specific, with productivity being the only one that is both widely shared across tasks and challenging."

W2: The closest testbeds are not compared against. Compositional-ARC ... ExeDec ... ARC-AGI-2 ..., Shojaee et al. ... and PlanBench ... is absent too, and no LLM is run on TranSGrid. Whether the diagnosis holds beyond 88M-parameter models trained from scratch is open.

Thank you for these references. We compare TranSGrid with Compositional-ARC and ExeDec, and report LLM results, in Q3. The remaining works and results of larger models are discussed below.

Reasoning-model puzzles. Compared with ARC-AGI-2, Tower of Hanoi and related puzzles (Shojaee et al.), and PlanBench, TranSGrid has two advantages for studying systematic generalization.

  • Beyond productivity. In Tower of Hanoi and related puzzles and in PlanBench, problem instances vary mainly in size, i.e., the number of objects and the solution length (Shojaee et al., Sec. 3.1; PlanBench, Sec. 6). Generalizing along this axis mainly tests productivity, which existing benchmarks already share. TranSGrid additionally tests inductive and abductive reasoning beyond productivity.
  • Controlled reasoning demands. ARC-AGI-2 also involves interacting rules, but each task is unique and its difficulty is measured by human solve rates, so it does not separate or precisely control the inductive and abductive demands. TranSGrid computes the deductive, inductive, and abductive loads of each instance (LL, KK, AabdA_{\mathrm{abd}}) and can reduce the inductive or abductive load while keeping the other factors fixed, so failures can be attributed to specific reasoning demands.

We therefore see these benchmarks as complementary: they measure overall problem-solving ability, whereas TranSGrid isolates and separately evaluates the inductive and abductive demands that systematic generalization requires. We have added a discussion of these works to Sec. 2.

Beyond 88M parameters. We trained two models larger than ID 7 with the same training setup. And the results are listed below.

Solve rate (%), Greedy / Top@8:

ID (Params)TestTranSGridEasyMediumHard
7 (88.43M)79.63 / 80.1755.31 / 61.6095.94 / 98.3854.25 / 65.1915.75 / 21.25
8 (353.02M)79.69 / 80.1757.56 / 63.8896.81 / 98.4458.13 / 70.0017.75 / 23.19
9 (470.59M)78.94 / 79.3554.56 / 60.6795.81 / 98.2552.75 / 63.8815.13 / 19.88

Scaling up beyond ID 7 does not help close the gap between Test and TranSGrid: ID 8 improves only slightly, and ID 9 even performs slightly worse than IDs 7 and 8.

We have added these results to Appendix G.

AuthorsOpenReview ↗

Response to Reviewer TRJ4 (2/4)

W3: The reasoning labels add little. ... The curves are never separated from each other or from L. The gold-prefix variant leaves a one-step choice among 86 actions, and "not due to shorter output" has no control ... the question is which shift, and the labels do not answer it.

Thank you for your constructive feedback. We will address your points with two analyses, one holding length fixed and one separating the two reasoning loads with matched controls, and then clarify the gold-prefix variant of TranSGrid (SCAN).

Separating each load from LL. The table below reports solve rates (%) of ID 7 (greedy) across inductive loads KK and abductive loads AA at each fixed reference length LL (lengths 10–12 are omitted as they additionally involve productivity, and lengths 1–2 as they contain only K=0K=0 and A=1A=1).

LLK=0K=1K=2K=3A=1A=2A=3A=4
3100.0097.31––97.75–––
4100.0083.28––89.6781.82––
5100.0076.29––84.2476.65––
694.4477.4243.24–90.7043.2361.70–
793.7573.0839.39–91.3044.4440.13–
890.0058.8236.31–66.0442.6235.2039.75
991.4350.0020.938.5075.5131.2528.5710.59

At each fixed LL, solve rates decrease monotonically as KK increases, and decrease as the abductive level increases in all but two cells (bold). Both loads therefore could lower performance even when length is fixed.

We have added these results to Sec. 4.4.

Separating the two loads with matched controls: If we understand correctly, by "which shift" you mean which property of TranSGrid causes the gap from Test; please let us know if we have misunderstood your point. To test whether each label captures a distinct demand, we built two matched controls from the original instances. Each reduces one demand while keeping the other factors fixed, and both modify only lengths 1–9, since lengths 10–12 additionally involve productivity.

  • TranSGrid (Inductive--): We keep the initial board and replace only the induction-rule spans with random actions, so that no induction rule remains. Each instance keeps an identical length, abductive score, and number of changed cells.
  • TranSGrid (Abductive--): We keep the initial board and the action types, order, and induction rules, and change only the positions of actions, moving each unit (an induction rule or a standalone action) as a whole to minimize the spatial overlap between units. This lowers the abductive score in 2,086 of 3,600 instances.

Solve rate (%), Greedy / Top@8; Δ is relative to TranSGrid.

IDTestTranSGridInductive--ΔAbductive--Δ
125.81 / 36.3816.10 / 20.2521.19 / 32.42+5.09 / +12.1716.67 / 20.90+0.57 / +0.65
235.79 / 51.5418.10 / 23.7727.06 / 43.15+8.96 / +19.3818.38 / 23.90+0.28 / +0.13
361.85 / 64.1226.83 / 33.0250.02 / 55.63+23.19 / +22.6126.94 / 32.81+0.11 / −0.21
471.27 / 72.5032.06 / 40.8557.69 / 63.02+25.63 / +22.1731.98 / 39.77−0.08 / −1.08
573.90 / 75.1537.73 / 47.9061.54 / 66.79+23.81 / +18.8938.40 / 49.00+0.67 / +1.10
678.33 / 78.9050.40 / 58.5668.58 / 72.44+18.18 / +13.8857.25 / 63.21+6.85 / +4.65
779.63 / 80.1755.31 / 61.6070.85 / 74.44+15.54 / +12.8461.75 / 66.31+6.44 / +4.71
  • Each factor matters on its own. Reducing either induction or abduction while holding the other fixed improves performance.
  • Removing induction yields the largest gains (up to +25.63), but no model reaches its Test performance.
  • Reducing abduction mainly helps larger models. IDs 6–7 gain +6.44–6.85 (greedy) and +4.65–4.71 (Top@8), while IDs 1–5 change by at most ±1.10. Further analysis shows that, since Abductive-- keeps every induction rule intact, smaller models remain bottlenecked by induction, whereas larger models solve many previously failed instances once abduction is reduced.

TranSGrid (SCAN). This variant was initially designed to reproduce the action-explicit goals of benchmarks such as SCAN. It provides the first L−1L-1 gold actions, but the task does not reduce to predicting a single action: the model still needs to execute all L−1L-1 given actions to track the board state before it can predict the final one. However, we acknowledge that this setting also introduces factors beyond the abductive demand.

In the revised paper, we have replaced the TranSGrid (Decoupled) and TranSGrid (SCAN) experiments with these two matched controls (Sec. 5.1–5.2).

AuthorsOpenReview ↗

Response to Reviewer TRJ4 (3/4)

Q1: Can the same seven architectures be trained and evaluated on one existing benchmark, gSCAN or COGS, to show that it fails to separate what TranSGrid separates? If not, the claims should be scoped to the testbed.

Thank you for this suggestion. We trained the same seven models on the original COGS training set and evaluated them on its test and generalization sets. Accuracy (%) is listed below.

IDCOGS TestCOGS GenGen (Lexical)Gen (Structural)TranSGrid TestTranSGrid
199.5063.0773.411.0725.8116.10
299.8066.2077.021.2735.7918.10
399.8765.4176.180.8361.8526.83
499.8758.1367.770.3371.2732.06
599.8760.0670.000.4073.9037.73
699.8748.8156.940.0778.3350.40
799.8760.3370.360.2079.6355.31

COGS does not separate the seven models, or their reasoning demands, in the way TranSGrid does. All seven models are at ceiling on the COGS test set (99.50–99.87) and near zero on its structural cases (0.07–1.27). Lexical accuracy does not increase with model size. In contrast, TranSGrid orders all seven models monotonically on both Test and TranSGrid. Moreover, COGS divides its generalization set by generalization type (lexical vs. structural), whereas TranSGrid quantifies each reasoning load at fine-grained levels (L=1L = 1–1212, K=0K = 0–33, and AabdA_{\mathrm{abd}} levels 1–5) and can vary each one while keeping the other factors fixed.

We have added these results to Appendix E.

Q2: At fixed concealment level, does solve rate still fall with K? A joint table of K against the abductive level would show whether the two loads are separable.

Thank you for the suggestion. The table below reports ID 7 (greedy) solve rates (%) by inductive load KK and abductive level AA within the training range (L≤9L \le 9). Numbers of instances are given in parentheses, and "–" indicates cells with fewer than 10 instances.

A=1A=2A=3A=4
K=099.45 (1,095)90.91 (22)––
K=187.63 (679)77.41 (425)54.55 (11)–
K=210.00 (10)38.24 (387)40.25 (487)38.17 (186)
K=3–––8.50 (294)

At each abductive level, solve rates decrease as KK increases. Together with the matched controls in W3, these results suggest that the two loads can be separated.

We hope this addresses your question. We have added this table to Appendix D.

Reviewer_5joJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines the limitations of existing systematic generalization benchmarks from a reasoning perspective. It identifies three common simplifications: elemental composition, productivity-based evaluation, and action-explicit goals. The authors introduce TranSGrid, a controllable grid-transformation benchmark that combines deductive, inductive, and abductive reasoning. Across seven Transformer models, performance drops substantially relative to the standard held-out test set, even within the training sequence-length range. Two controlled variants show that reducing action interactions or providing explicit action information improves performance. The authors argue that evaluating systematic generalization requires considering all three forms of reasoning, rather than focusing mainly on productivity.

Critical strengths

1.The distinction between deductive, inductive, and abductive reasoning offers a useful way to examine systematic generalization. It helps clarify which reasoning demands existing benchmarks capture and which they may overlook. 2. TranSGrid is flexible and allows solutions to be verified automatically. The experiments across seven Transformer models and the controlled variants show clear performance gaps, suggesting that standard held-out evaluations can miss important difficulties in systematic generalization.

Critical weaknesses
  1. The paper uses reference sequence length, the number of predefined induction rules, and action concealment as proxies for deductive, inductive, and abductive reasoning, respectively. However, it does not independently validate whether these proxies capture the intended reasoning abilities. They may also reflect broader differences in task complexity, and the inductive and abductive measures may be confounded with sequence length. This makes it difficult to attribute the performance drops to specific reasoning demands.
  2. The TranSGrid (SCAN) variant gives the model the first L−1L-1 ground-truth actions, leaving only the final action to predict. This substantially reduces both the prediction space and the opportunity for errors to accumulate. The improvement therefore does not isolate the effect of reducing abductive reasoning demands, leaving a central part of the paper’s interpretation insufficiently supported.
Questions for authors
  1. What evidence supports interpreting the proposed inductive and abductive difficulty measures as measures of those reasoning abilities, rather than general task complexity?
  2. Could the authors compare performance across inductive and abductive difficulty levels while holding reference sequence length fixed?
  3. How could the authors separate the effect of reduced abductive reasoning demands from the benefits of ground-truth action prefixes, including shorter prediction lengths and less error accumulation?
  4. Since multiple action sequences can lead to the same target board, how do the authors establish that difficulty measured from a reference sequence reflects the difficulty of solving the task?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 5joJ (1/2)

Thank you for your constructive feedback and support for our work. We address specific weaknesses and questions below.

W1 & Q1: What evidence supports interpreting the proposed inductive and abductive difficulty measures as measures of those reasoning abilities, rather than general task complexity?

Thank you for this insightful question. TranSGrid (Decoupled) and TranSGrid (SCAN) were initially designed to reproduce two simplifications common in prior benchmarks: elemental composition and action-explicit goals. On closer examination, however, we found that reproducing them in this way changes more than one difficulty factor, so their gains cannot be attributed to a single reasoning demand. To separate each demand from general task complexity, we therefore built two controlled variants directly from the original TranSGrid instances. Each reduces one demand while keeping the other factors fixed, and both modify only lengths 1–9, since lengths 10–12 additionally involve productivity.

  • TranSGrid (Inductive--): We keep the initial board and replace only the induction-rule spans with random actions, so that no induction rule remains. Each instance keeps an identical length, abductive score, and number of changed cells.
  • TranSGrid (Abductive--): We keep the initial board and the action types, order, and induction rules, and change only the positions of actions, moving each unit (an induction rule or a standalone action) as a whole to minimize the spatial overlap between units. This lowers the abductive score in 2,086 of 3,600 instances.

Solve rate (%), Greedy / Top@8; Δ is relative to TranSGrid.

IDTestTranSGridInductive--ΔAbductive--Δ
125.81 / 36.3816.10 / 20.2521.19 / 32.42+5.09 / +12.1716.67 / 20.90+0.57 / +0.65
235.79 / 51.5418.10 / 23.7727.06 / 43.15+8.96 / +19.3818.38 / 23.90+0.28 / +0.13
361.85 / 64.1226.83 / 33.0250.02 / 55.63+23.19 / +22.6126.94 / 32.81+0.11 / −0.21
471.27 / 72.5032.06 / 40.8557.69 / 63.02+25.63 / +22.1731.98 / 39.77−0.08 / −1.08
573.90 / 75.1537.73 / 47.9061.54 / 66.79+23.81 / +18.8938.40 / 49.00+0.67 / +1.10
678.33 / 78.9050.40 / 58.5668.58 / 72.44+18.18 / +13.8857.25 / 63.21+6.85 / +4.65
779.63 / 80.1755.31 / 61.6070.85 / 74.44+15.54 / +12.8461.75 / 66.31+6.44 / +4.71

Removing the induction rules improves all models, although length, abductive score, and the number of changed cells are identical. Reducing abduction, with the action types, order, and induction rules unchanged, mainly helps the larger models (IDs 6–7), since smaller models remain bottlenecked by the intact induction rules. These distinct profiles suggest that the two measures capture different sources of difficulty rather than a single notion of task complexity.

We have revised Sec. 5.1–5.2 accordingly.

W1 & Q2: Could the authors compare performance across inductive and abductive difficulty levels while holding reference sequence length fixed?

Thank you for the suggestion. The table below reports solve rates (%) of ID 7 (greedy) across inductive loads KK and abductive loads AA at each fixed reference length LL within the training range. Rows with L≤2L \le 2 are omitted, as they contain only K=0K=0 and A=1A=1.

LLK=0K=1K=2K=3A=1A=2A=3A=4
3100.0097.31––97.75–––
4100.0083.28––89.6781.82––
5100.0076.29––84.2476.65––
694.4477.4243.24–90.7043.2361.70–
793.7573.0839.39–91.3044.4440.13–
890.0058.8236.31–66.0442.6235.2039.75
991.4350.0020.938.5075.5131.2528.5710.59

At each fixed LL, solve rates decrease monotonically as KK increases, and decrease as the abductive level increases in all but two cells (bold; see Q4 below for detailed analysis).

We have added these results to Sec. 4.4.

AuthorsOpenReview ↗

Response to Reviewer 5joJ (2/2)

W2 & Q3: How could the authors separate the effect of reduced abductive reasoning demands from the benefits of ground-truth action prefixes, including shorter prediction lengths and less error accumulation?

Thank you for this insightful observation. TranSGrid (SCAN) was initially designed to reproduce the action-explicit goals of benchmarks such as SCAN, where the input specifies the complete action sequence and the model only needs to parse or execute it (e.g., jump opposite left and walk → LTURN LTURN JUMP WALK). Likewise, TranSGrid (SCAN) provides the first L−1L-1 gold actions, but the task does not reduce to predicting a single action: the model still needs to execute all L−1L-1 given actions to track the board state before it can predict the final one.

However, we acknowledge that this setting also introduces factors beyond the abductive demand. To better isolate this reasoning form, we built TranSGrid (Abductive--), which only changes the positions of the units in each reference answer to reduce their overlap (see details in our response to Q1).

In the revised paper, we have replaced the TranSGrid (SCAN) experiment with this matched control (Sec. 5.2).

Q4: Since multiple action sequences can lead to the same target board, how do the authors establish that difficulty measured from a reference sequence reflects the difficulty of solving the task?

Thank you for this insightful observation. Using the reference sequence is a deliberate trade-off between accuracy and cost: AabdA_{\mathrm{abd}} is accurate enough in most cases at negligible cost, whereas the shortest solution is far more expensive.

Accuracy. In the abductive columns of the table in Q2, solve rates decrease as the abductive level increases at each fixed LL, except in two cells (bold). Compared with the preceding level, these cells contain a larger share of successful predictions shorter than the reference, which partly reflects your concern: when shorter solutions exist, AabdA_{\mathrm{abd}} can overestimate difficulty. Since such exceptions occur in only two cells, AabdA_{\mathrm{abd}} computed from the reference is already accurate enough in most cases.

Cost. AabdA_{\mathrm{abd}} needs only the reference length and the number of changed cells, whereas exact shortest-solution search grows exponentially with sequence length and rapidly with board size (about 2.6×10172.6\times10^{17} candidate sequences up to length 9 on a 6×66\times6 board), which would prevent the score from scaling to longer sequences and larger boards.

We have added this analysis to Appendix C, where we also discuss this limitation.

We greatly appreciate the reviewer's feedback and welcome further discussions

Dear Reviewer 5joJ,

We sincerely appreciate the time and effort you have dedicated to reviewing our work!

As the discussion period draws to a close, we would greatly appreciate it if you could take a moment to review our responses and let us know whether they adequately address your concerns and questions.

Best regards,

The Authors

Reviewer_GUxwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks what existing systematic generalization tasks may miss. The authors introduce TranSGrid, where a model sees an initial grid and a target grid and must output actions that transform one into the other. The answer is checked by executing the actions, so different valid solutions can be accepted.

Seven encoder–decoder Transformers are tested. The largest model solves 79.63% of the ordinary Test set, but 55.31% of TranSGrid and only 15.75% of its Hard subset with greedy decoding. The paper also studies two easier variants: one reduces interactions between actions, and one gives the model almost all gold actions.

Critical strengths
  1. TranSGrid has a clear action space and an executable correctness check. Accepting any valid action sequence is a good design choice.

  2. The paper tests seven model sizes and shows that the Test–TranSGrid gap remains for some sequence lengths seen during training. This supports the point that testing only longer sequences is insufficient.

  3. The two task variants are useful first steps toward understanding which parts make the task hard.

Critical weaknesses
  1. The proposed “inductive load” is strongly mixed with sequence length. Appendix Table 7 shows that cases with three induction rules occur only at lengths 9–12, while cases with zero rules are concentrated at short lengths. Figure 3a compares performance across rule counts, but this cannot separate the effect of rule interactions from the effect of length. Showing that many cases at K=2K=2 are within the training length range does not fully solve this issue: length 8 can still be harder than length 3. Please compare different KK values at the same LL, with enough examples in each group.

  2. The easier variants change more than one difficulty factor. TranSGrid (Decoupled) reduces action interactions, but it may also make the target easier to read and the solution easier to search for. TranSGrid (SCAN) supplies the first L−1L-1 gold actions and leaves only the final action to predict. Its improvement may therefore come from the much shorter prediction problem as well as reduced abduction. More closely matched controls are needed before assigning each gain to one reasoning form.

  3. The abductive score is a proxy tied to reference length. It uses L−LeffL-L_{\mathrm{eff}}, where LL is the length of one reference solution. The same grid pair may have a shorter valid solution, which the paper’s own evaluation would accept. In such a case, the score can reflect redundancy in the sampled reference rather than a real need for hidden-action inference. Please examine shortest-solution length, or at least report how often shorter solutions exist.

  4. The generalization claim needs a clearer boundary. The experiments use synthetic 6×6 grids and purpose-trained Transformers of at most 88.43M parameters. The evaluation instances are held out, but the paper does not test generalization to held-out types of composition rules. I would describe the result as strong evidence about this testbed, and be more cautious when concluding what existing systematic generalization benchmarks generally miss.

Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer GUxw (1/3)

Thank you for your comments and constructive suggestions. We address specific weaknesses and questions below.

W1: The proposed "inductive load" is strongly mixed with sequence length. ... Please compare different KK values at the same LL, with enough examples in each group.

Thank you for your insightful suggestion. We reanalyzed the existing TranSGrid results by comparing different inductive loads KK at each fixed reference length LL. The table below reports the results of model ID7 under greedy decoding for L=1–9L=1–9 (lengths 10–12 are omitted as they additionally involve productivity). Each cell shows solved/total (accuracy), and "--" indicates that no examples are available.

LK=0K=1K=2K=3
1400/400 (100.00%)------
2400/400 (100.00%)------
366/66 (100.00%)325/334 (97.31%)----
465/65 (100.00%)279/335 (83.28%)----
571/71 (100.00%)251/329 (76.29%)----
634/36 (94.44%)24/31 (77.42%)144/333 (43.24%)--
715/16 (93.75%)19/26 (73.08%)141/358 (39.39%)--
827/30 (90.00%)20/34 (58.82%)122/336 (36.31%)--
932/35 (91.43%)14/28 (50.00%)9/43 (20.93%)25/294 (8.50%)

Our findings indicate that at every fixed LL, the solve rate decreases monotonically as KK increases. For example, at L=9L = 9, it drops from 91.43% to 50.00%, 20.93%, and 8.50% for K=0,1,2,3K = 0, 1, 2, 3, respectively.

We have added these results to Sec. 4.4.

W2: The easier variants change more than one difficulty factor. ... More closely matched controls are needed before assigning each gain to one reasoning form.

Thank you for this insightful feedback. TranSGrid (Decoupled) and TranSGrid (SCAN) were initially designed to reproduce two simplifications common in prior benchmarks: elemental composition and action-explicit goals. We agree that reproducing them in this way changes more than one difficulty factor. To isolate induction and abduction, we built two controlled variants directly from the original TranSGrid instances. Both modify only lengths 1–9, since lengths 10–12 additionally involve productivity.

  • TranSGrid (Inductive--): We keep the initial board and replace only the induction-rule spans with random actions, so that no induction rule remains. Each instance keeps an identical length, abductive score, and number of changed cells.
  • TranSGrid (Abductive--): We keep the initial board and the action types, order, and induction rules, and change only the positions of actions. Each unit (an induction rule or a standalone action) is moved as a whole to minimize the spatial overlap between units. This lowers the abductive score in 2,086 of 3,600 instances.

Solve rate (%), Greedy / Top@8; Δ is relative to TranSGrid.

IDTestTranSGridInductive--ΔAbductive--Δ
125.81 / 36.3816.10 / 20.2521.19 / 32.42+5.09 / +12.1716.67 / 20.90+0.57 / +0.65
235.79 / 51.5418.10 / 23.7727.06 / 43.15+8.96 / +19.3818.38 / 23.90+0.28 / +0.13
361.85 / 64.1226.83 / 33.0250.02 / 55.63+23.19 / +22.6126.94 / 32.81+0.11 / −0.21
471.27 / 72.5032.06 / 40.8557.69 / 63.02+25.63 / +22.1731.98 / 39.77−0.08 / −1.08
573.90 / 75.1537.73 / 47.9061.54 / 66.79+23.81 / +18.8938.40 / 49.00+0.67 / +1.10
678.33 / 78.9050.40 / 58.5668.58 / 72.44+18.18 / +13.8857.25 / 63.21+6.85 / +4.65
779.63 / 80.1755.31 / 61.6070.85 / 74.44+15.54 / +12.8461.75 / 66.31+6.44 / +4.71
  • Each factor matters on its own. Reducing either induction or abduction while holding the other fixed improves performance.
  • Removing induction yields the largest gains (up to +25.63), but no model reaches its Test performance.
  • Reducing abduction mainly helps larger models. IDs 6–7 gain +6.44–6.85 (greedy) and +4.65–4.71 (Top@8), while IDs 1–5 change by at most ±1.10. Further analysis shows that, since Abductive-- keeps every induction rule intact, smaller models remain bottlenecked by induction, whereas larger models solve many previously failed instances once abduction is reduced.

In the revised paper, we have replaced the TranSGrid (Decoupled) and TranSGrid (SCAN) experiments with these two matched controls (Sec. 5.1–5.2).

AuthorsOpenReview ↗

Response to Reviewer GUxw (2/3)

W3: The abductive score is a proxy tied to reference length. ... In such a case, the score can reflect redundancy in the sampled reference rather than a real need for hidden-action inference. Please examine shortest-solution length, or at least report how often shorter solutions exist.

Thank you for this insightful observation. Using the reference length is a deliberate trade-off between accuracy and cost: AabdA_{\mathrm{abd}} is accurate enough in most cases at negligible cost, whereas the shortest-solution length is far more expensive.

How often shorter solutions exist. Across the valid solutions produced by our seven models under both decoding strategies, at least 898 of the 4,800 instances (18.71%) admit a solution shorter than the reference. However, the shortest length is also a proxy, because models do not always find the shortest solution: at least 6.69% of their successful predictions are longer than another valid solution to the same instance.

Accuracy. The table below reports solve rates (%) of ID 7 (greedy) by LL and abductive level within the training range (L≤3L \le 3 contains only level 1 and is omitted).

LLA=1A=2A=3A=4
489.6781.82––
584.2476.65––
690.7043.2361.70–
791.3044.4440.13–
866.0442.6235.2039.75
975.5131.2528.5710.59

Within each LL, solve rates decrease as the abductive level increases, except in two cells (bold). Compared with the preceding level, these cells contain a larger share of successful predictions shorter than the reference, which partly reflects your concern: when shorter solutions exist, AabdA_{\mathrm{abd}} can overestimate difficulty.

Cost. AabdA_{\mathrm{abd}} needs only the reference length and the number of changed cells, whereas computing the exact shortest-solution length requires a search that grows exponentially with sequence length and rapidly with board size (about 2.6×10172.6\times10^{17} candidate sequences up to length 9 on a 6×66\times6 board), which would prevent the score from scaling to longer sequences and larger boards.

We have added the accuracy and cost analysis to Appendix C.

AuthorsOpenReview ↗

Response to Reviewer GUxw (3/3)

W4: The generalization claim needs a clearer boundary. ... the paper does not test generalization to held-out types of composition rules. I would describe the result as strong evidence about this testbed, and be more cautious when concluding what existing systematic generalization benchmarks generally miss.

Thank you for this suggestion. If we understand correctly, by "held-out types of composition rules" you mean composition rules that are held out from training. TranSGrid already evaluates this kind of generalization. Models are trained only on randomly sampled action sequences, and the 12 induction rules used to construct TranSGrid are never deliberately included. In a random sample of 1M training sequences, only 0.0045% contain any rule instance, and 5 of the 12 rules never appear. The rules in TranSGrid, and therefore their combinations, are essentially held out from training. Please let us know if we have misunderstood your point.

Regarding the scope of our claims, we also trained the same seven models on COGS, a representative existing benchmark, to see how its evaluation of these models differs from that of TranSGrid.

IDCOGS TestCOGS GenGen (Lexical)Gen (Structural)TranSGrid TestTranSGrid
199.5063.0773.411.0725.8116.10
299.8066.2077.021.2735.7918.10
399.8765.4176.180.8361.8526.83
499.8758.1367.770.3371.2732.06
599.8760.0670.000.4073.9037.73
699.8748.8156.940.0778.3350.40
799.8760.3370.360.2079.6355.31

All seven models are at ceiling on the COGS test set and near zero on its structural cases, and lexical accuracy does not increase with model size. In contrast, TranSGrid orders all seven models monotonically. Moreover, COGS divides its generalization set by generalization type (lexical vs. structural), whereas TranSGrid quantifies each reasoning load at fine-grained levels (L=1–12L = 1–12, K=0K = 0–33, and AabdA_{\mathrm{abd}} levels 1–5) and can vary each one while keeping the other factors fixed. COGS therefore does not separate the seven models, or their reasoning demands, in the way TranSGrid does.

Beyond the seven trained models, we also evaluated two reasoning LLMs, DeepSeek V4.1 Flash 1017 and GLM-5.3-Flash, on balanced 600-instance subsets of TranSGrid and two recent benchmarks, Compositional-ARC and ExeDec.

Accuracy (%) and average number of output tokens per instance:

DatasetSubsetDeepSeek Acc.DeepSeek Avg. TokensGLM Acc.GLM Avg. Tokens
Compositional-ARCColor98.8399.33
Shape96.3321,29191.6759,427
Exact Match94.0082.17
ExeDecLength Generalization71.67209,98375.83120,277
Compose Different Concepts82.50160,99785.8353,144
Switch Concept Order86.67101,36785.0079,329
Compose New Operation74.17217,85874.17133,911
Add Operation Functionality86.67107,73383.33119,571
TranSGridEasy91.0030,14596.5026,990
Medium55.50166,96783.50123,827
Hard21.50320,60658.00394,722

Both LLMs perform well on the two benchmarks, with exact-match accuracies of 94.00% and 82.17% on Compositional-ARC and 71.67–86.67% and 74.17–85.83% across the five ExeDec tasks, showing that current reasoning LLMs can already largely solve them. In contrast, TranSGrid remains challenging for the same LLMs: they solve most Easy instances (91.00% and 96.50%) but only 21.50% and 58.00% of Hard instances, while generating 320,606 and 394,722 output tokens per instance on average, many more than on Compositional-ARC and on any ExeDec task. Together with the COGS comparison, these results suggest that TranSGrid captures difficulties not reflected in these existing benchmarks, for both purpose-trained models and current reasoning LLMs. Moreover, since TranSGrid is procedurally generated, its difficulty can be further increased with larger boards and longer action sequences at almost no cost, so it can remain challenging as models continue to improve.

We have added the COGS and LLM results to Appendices E and H, respectively.

We greatly appreciate the reviewer's feedback and welcome further discussions

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion summary

We sincerely thank the PCs, SAC, and AC for facilitating the discussion, and all reviewers for their thorough evaluations and constructive suggestions, which have greatly improved our paper.

We are pleased that the reviewers have recognized:

  • Reasoning-centered lens (Reviewer 5joJ). Distinguishing deductive, inductive, and abductive reasoning offers a useful way to examine systematic generalization.
  • Flexible and well-designed testbed that accepts multiple correct solutions (All Reviewers). TranSGrid is a well-designed and flexible testbed for systematic generalization, and its automatic verification accepts any correct action sequence, not only the reference one.
  • Well-designed controlled variants (All Reviewers). The two controlled variants are a useful step toward understanding what makes systematic generalization difficult.
  • Gap within the training length range (All Reviewers). The Test-TranSGrid gap persists within the training length range, providing solid evidence for our claim that productivity alone is insufficient to evaluate systematic generalization.

Throughout the discussion phase, we have addressed the following main concerns raised by the reviewers:

  1. Fixed-length analysis of inductive and abductive reasoning (All Reviewers). We reanalyzed the existing TranSGrid results and show that at each fixed reference length, solve rates still decrease as either the inductive or the abductive load increases.
  2. The two original controlled variants (All Reviewers). We replaced TranSGrid (Decoupled) and TranSGrid (SCAN) with TranSGrid (Inductive--) and TranSGrid (Abductive--), each reducing one demand while keeping length and other demands fixed. The results show that reducing either demand alone improves performance.
  3. Reference length for the abductive score (Reviewers GUxw, 5joJ). Our analysis shows that computing the score from the reference length is accurate in most cases at negligible cost, whereas exact shortest-solution search grows exponentially with sequence length.
  4. Comparison with existing benchmarks (Reviewers GUxw, TRJ4). We trained the same seven models on COGS and evaluated two reasoning LLMs on Compositional-ARC, ExeDec, and TranSGrid. The results show that COGS does not separate the seven models as TranSGrid does, and that LLMs largely solve Compositional-ARC and ExeDec but still struggle with TranSGrid Hard instances while using many more output tokens.
  5. Larger models (Reviewer TRJ4). Scaling to 353.02M and 470.59M parameters does not help close the gap between Test and TranSGrid.

We are glad that both Reviewer 5joJ and Reviewer TRJ4 agree that the additional experiments and responses have addressed most of their concerns, many of which are also shared by Reviewer GUxw. We are also delighted that Reviewer TRJ4 recognized that our experiments are sound and well support the core claim of this paper.

Once again, we thank the PCs, SAC, AC, and all reviewers for their time and efforts in reviewing the paper and helping us improve it! We have uploaded a revised PDF of the paper, where revisions are marked in blue for easy identification.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.