acceptodds
Under review as a conference paper at ICLR 2027

Thought without systematicity? Evaluating reasoning models on rule induction tasks

Abstract

A central tenet of human cognition is systematicity, the principle that understanding one concept is inherently tied to understanding close variations of that concept. Do reasoning models robustly exhibit such systematicity? If so, we would expect consistent performance on structurally equivalent variants of the same task. Here, we extend established rule induction tasks from cognitive science to assess the systematicity of thought in current reasoning models. Each task family has compositional structure that we use to create structurally equivalent task variations through task isomorphisms such as recombination and substitution. We find that despite being able to correctly solve a task, models often fail on structurally equivalent variants of the same task. These findings suggest that many model behaviors lack systematicity, rendering it difficult to robustly establish the cognitive abilities of reasoning models beyond the particular contexts they were evaluated in.

Primary area:
datasets and benchmarks
Keywords:
systematicity of thought, reasoning models, rule induction, cognitive science, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_H6n4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether reasoning models perform consistently across structurally equivalent tasks. It focuses on structure-preserving task isomorphisms (symbol substitution, feature rebinding, constituent permutation and example reordering). Four families of problems are considered: grammar-based instruction-learning, symbolic Raven matrices, program induction over integer sequences and Boolean concept learning. For each task, the authors generate 5 variants per type of invariance. A variant counts as solved if at least 3 of 5 attempts answer correctly the entire query set. Results are reported for Gemini 3.1 (Flash-Lite, Flash and Pro), Claude Opus 4.7 and GPT-5.5, using 3 metrics: whether any variant is solved, the fraction solved, and whether all are solved, averaged over variation sets. Three task families show substantial gaps, which the authors interpret as a lack of systematicity. Further experiments analyze sampling noise, greedy decoding, adding hints, changing task difficulty and varying reasoning effort.

Critical strengths
  • The problem and task setup are clear. Synthetic variant generation through isomorphisms makes it explicit what differs between variants, and the approach is general enough to be applied to other tasks.

  • The distinction between within-task-variation noise (model stochasticity) and between-task-variation variance (task variations) is important. The variance decomposition analysis helps separating estimated differences between variants from stochastic noise across attempts. This helps explaining what contributes to the observed gap.

Critical weaknesses
  • The gap between solving at least one variant and solving all variants can arise from both stochastic unreliability and changes in performance when the task is reformulated. In Appendix A these sources of variance are usefully separated, with little estimated between-variant variance in several conditions. The authors also state that systematicity requires reliability of performance. The paper should make clear in the main discussion where the evidence supports sensitivity to task transformation and where it reflects unreliable performance. The zero-temperature Gemini experiment provides additional evidence of sensitivity to task reformulation, but does not quantify the influence of noise under default sampling.

  • Even if only isomorphisms are considered, some variants may not be equally difficult, but the analysis still assumes they are. In Section 4.2, the paper reports the permuted Raven example as "significantly more difficult because the features are no longer spatially aligned across columns". The Raven task family also has some of the largest between-variant variance estimates (Table 1). This makes it difficult to interpret the gap as a comparison between equally difficult tasks, even with the transformations being valid isomorphisms.

More generally, the task generators and controlled transformations are useful, but the potential analysis of the systematicity remains still limited. In the related work it is acknowledged that sensitivity to task reformulations is already well studied. The paper needs to explain better what effects persist beyond sampling noise and difficulty differences, to understand how these rule induction tasks aid in the understanding of systematicity.

Minor point: in the example in Section 4.3 the query has length equal to one sequence in the support set, which is in contradiction with what is written in the "Support and query set" paragraph. This may concern only the example.

Questions for authors
  • If each variant of a task has the same success rate (per-attempt), what gap is expected and how does it compare with the observed gap once one accounts for the uncertainty?
  • At zero sampling temperature, do repeated attempts of the same task variant give the same answer with Gemini?
  • Regarding the Raven's problem, what happens if the analysis separates between aligned and misaligned variants? Do differences between variants persist when alignment is consistent?
  • How many distinct base tasks and variation sets are considered per family? Were multiple variation sets generated from same base task? These should be reported in the main text.
  • Please report confidence intervals accounting for variants from same base task.
  • With structured output, do failures due to formatting occur? If so, how often do they occur and how are they scored?

For future work: an analysis of the reasoning traces of failed variants could be helpful.

6 comments

Thank you for taking the time to carefully review our paper. We have run several additional experiments and analyses following your suggestions that we think have improved the paper. We address your specific comments point-by-point below.

Weakness 1: Clarify variance decomposition

Question 1: If each variant of a task has the same success rate (per-attempt), what gap is expected and how does it compare with the observed gap once one accounts for the uncertainty?

(Please allow us to copy our response to Reviewer eKti who raised a very similar point)

We agree that it is important to distinguish transformation sensitivity from task-level unreliability to contextualize what is driving the observed systematicity gaps. We attempt to do so in Section 5.2 with the variance decomposition and greedy decoding analysis and we extend both following your and Reviewer eKti’s suggestions in the following:

  1. We now additionally perform a statistical test against a null model to determine when the between task variation is statistically significant. Concretely, to test the null hypothesis that all variations of a base task share the same success probability, we compute a stratified Pearson's chi-squared homogeneity statistic across task variations and use a permutation test that shuffles attempt outcomes across variations of each base task to obtain p-values. The results are in line with the between task variation variances we reported (we updated Table 1 in the PDF).

  2. We extend the greedy decoding experiments beyond Gemini models. Since neither GPT nor Claude reasoning models allow greedy decoding, we do so on current, open-weight reasoning models, Deepseek 4.1 Flash, GLM-5.3 and Kimi K3.1 Pro. Please kindly take a look at our general response to see the detailed results. Generally speaking, we observe similar gaps in systematicity in line with the findings on Gemini.

As we detail in the discussion, reliability is a prerequisite for strict systematicity and the strong within-task-variation variance plays a central role for the observed systematicity gaps. The greedy decoding analysis demonstrates that this lack of systematicity goes beyond ordinary stochastic sampling noise. We will revise the manuscript to make this distinction more prominently.

Weakness 2: Even if only isomorphisms are considered, some variants may not be equally difficult

We fully agree with you that structural equivalence formalized through isomorphisms does not by itself ensure equivalent inference difficulty. We carefully choose our task variations so as not to change their task difficulty. For the task variations on Raven you mention specifically, we ensure that all variations are permuted variants, i.e. we do not compare unpermuted to permuted instances. The corresponding statement in the submission was not making this point sufficiently clear and we will revise it accordingly.

Weakness 3: Better contextualize findings beyond prior work on robustness

We appreciate the suggestion and will revise the related work to more clearly emphasize the difference to prior work that evaluates robustness. Systematicity concerns structured changes that recombine familiar constituents and their structural roles (Fodor & Pylyshyn, 1988). We therefore consider data generating processes with compositional structure that allow us to create controlled compositional task transformations. This means that measuring systematicity is a type of robustness analysis but the inverse is not true: Perturbations by prior work do not systematically probe the reuse of constituents and operations across structured recombinations.

Minor point: in the example in Section 4.3 the query has length equal to one sequence in the support set

Thank you for pointing out this inconsistency in our example in Section 4.3, we will fix it in the paper.

Question 2: At zero sampling temperature, do repeated attempts of the same task variant give the same answer with Gemini?

We tested this for a subset of tasks and found the answer to be stable across attempts.

Question 3: Regarding the Raven's problem, what happens if the analysis separates between aligned and misaligned variants? Do differences between variants persist when alignment is consistent?

When we generate task variations in Raven, we ensure that all variations are permuted, i.e. we do not compare permuted to unpermuted variations. The corresponding statement in the submission was not making this point sufficiently clear and we will revise it accordingly.

Question 4: How many distinct base tasks and variation sets are considered per family? Were multiple variation sets generated from same base task? These should be reported in the main text.

We report the number of independent bases tasks per task family in the following table and will add this information to the manuscript. Note that we share a canonical task as the base task variation to obtain K=5 variations per transformation type.

Task familyBase tasksVariants per base taskAttemptsTotal tasks evaluated
Boolean98 (4 translation + 4 shuffle)5405
Raven104 (4 permutation)5250
Grammar1012 (4 translation + 4 recomposition + 4 shuffle)5650
Integer list128 (4 translation + 4 shuffle)5540

Question 5: Please report confidence intervals accounting for variants from same base task.

We added error bars reporting the standard error of the mean to the systematicity bullet plot. In terms of the confidence interval for the differences, we used the Wilcoxon signed-rank test to determine whether the systematicity gap is significant across tasks for each model.

ModelGap meanGap SEMWilcoxon p
Claude Opus 4.70.28750.06260.0004
GPT-5.50.16250.05000.0042
Gemini 3 Flash0.30420.06820.0004
Gemini 3.1 Flash-Lite0.19240.06370.0042
Gemini 3.1 Pro0.28750.06260.0004

Question 6: With structured output, do failures due to formatting occur? If so, how often do they occur and how are they scored?

Formatting failures occur rarely with the reasoning models considered here and when they do occur we retry until we get a valid response.

Thank you again for your constructive feedback that we believe helped improve our work. With the end of the discussion period coming closer, we wanted to kindly ask whether our response addresses your concerns? Please let us know if you have any further questions.

Reviewer_eKtiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether reasoning models exhibit systematicity, defined as the ability to consistently solve structurally equivalent variations of the same task. The authors formalize task equivalence through isomorphic transformations and adapt four rule-induction task families from cognitive science: grammar-based instruction learning, symbolic Raven's progressive matrices, program induction over integer sequences, and Boolean concept learning.

The evaluation considers five proprietary reasoning models and measures whether they can solve at least one, some, or all of five isomorphic task variations. The results show substantial gaps between solving at least one variation and solving all variations on several task families, particularly Raven's matrices and program induction. The authors further investigate stochastic decoding through variance decomposition and greedy decoding, and examine the effects of reasoning effort and additional task information. They conclude that current reasoning models often lack strict systematicity, raising concerns about how reliably performance on individual reasoning tasks generalizes to structurally equivalent contexts.

Critical strengths

1. A controlled approach to evaluating consistency beyond standard accuracy.

The most compelling aspect of this work is its use of task isomorphisms to evaluate reasoning consistency. Rather than relying on paraphrases or loosely defined perturbations, the authors construct variations with explicitly preserved input-output relationships. This provides a relatively principled way to examine whether successful reasoning transfers across different realizations of the same underlying rule. The adaptation of established cognitive tasks into procedurally generated evaluation families is also valuable, particularly because it allows the authors to examine several forms of compositional structure within a common framework.

2. The paper makes a meaningful effort to distinguish systematicity failures from stochastic decoding effects.

The authors recognize that inconsistency across task variations may arise from stochastic generation rather than sensitivity to the transformations themselves. The repeated evaluations, variance decomposition in Appendix A, and additional greedy-decoding experiments are therefore important components of the study. In particular, the persistence of systematicity gaps under zero-temperature decoding for the Gemini models provides evidence that at least part of the observed inconsistency cannot be attributed to ordinary sampling noise. This makes the empirical investigation more convincing than an analysis based solely on differences in accuracy across transformed inputs.

Critical weaknesses

1. The central systematicity metric does not sufficiently distinguish transformation sensitivity from ordinary task-level unreliability.

The paper places considerable emphasis on the gap between solving any and solving all task variations. However, this gap is inherently sensitive to a model's underlying success probability, even when that probability is identical across all isomorphic variations. For example, suppose each of the five variations has the same probability qq of passing the majority-vote criterion, with independent evaluations. The expected difference between the "any" and "all" indicators is 1−(1−q)5−q51-(1-q)^5-q^5. When q=0.5q=0.5, this difference is 0.9375, despite there being no variation-dependent difference in success probability. Thus, a substantial gap can arise without any sensitivity to the particular isomorphic transformation.

The authors are aware of stochasticity and explicitly investigate it in Section 5.2. Nevertheless, the variance decomposition in Table 1 makes the interpretation of the main results less straightforward than the discussion suggests. For instance, in integer-sequence program induction, Claude Opus 4.7 has estimated normalized between-variation variance components of 0.04 for substitution and 0.00 for reordering, while Section 5.1 highlights a 41.67-point gap between solving any and all variations. These findings are not contradictory, but they illustrate why the raw gap alone is insufficient evidence of substantial transformation-dependent behavior. The greedy-decoding experiments partially address this concern, but they are limited to the Gemini models. For the other models, the central result still mixes reliability under repeated sampling with sensitivity to task transformations. Reliability is certainly relevant to the authors' strict definition of systematicity, but these two phenomena should be distinguished when drawing conclusions about compositional reasoning.

I would find the central claim substantially more convincing if the reported gaps were compared against a null model in which all variations of a given base task have the same success probability. This would quantify the inconsistency attributable specifically to differences between task variations, rather than to finite-sample stochasticity alone.

2. The validity of the rule-induction tasks as unambiguous tests of systematicity is not sufficiently established.

The evaluation assumes that an incorrect prediction indicates a failure to infer or apply the underlying rule. This requires the support examples to identify the relevant query outputs sufficiently well. However, the paper does not establish this property equally rigorously across the four task families.

The authors explicitly address ambiguity in grammar-based instruction learning through specially constructed support examples, and in Boolean concept learning through an ideal-observer procedure. In contrast, the program-induction task samples programs from a compositional grammar and constructs support examples from randomly sampled input lists. Although the support and query lengths are separated to prevent a particular shortcut, this does not establish that the underlying program, or even its outputs on the query set, can be uniquely determined from the support examples. Distinct programs can agree on all observed examples while producing different outputs on unseen inputs. A related ambiguity can occur in Raven's matrices when multiple candidate rules agree on the observed rows. There is also an important distinction between formal structural equivalence and equivalent inference difficulty. The paper argues that isomorphic transformations preserve task structure, but the Raven example in Section 4.2 explicitly describes the feature-permuted variation as significantly harder because the learner must additionally infer feature correspondences. While sensitivity to such transformations is a legitimate object of study, the increased correspondence-search burden complicates the interpretation that the variations test the same reasoning ability under equal difficulty.

These issues matter because the paper's broader conclusions depend on treating success and failure across isomorphic tasks as evidence about systematic rule understanding. Without stronger guarantees of query-label identifiability and a more careful treatment of variation-induced difficulty, the observed failures cannot always be attributed specifically to a lack of systematicity.

Questions for authors

1. Separating stochastic unreliability from transformation sensitivity

Could the authors report the expected any-versus-all gap under a null hypothesis where all variations of a base task have identical success probabilities, estimated from that task's observed performance? How much of the reported gap remains after accounting for this baseline, particularly for the non-Gemini models that cannot be evaluated with greedy decoding? This would directly clarify the extent to which the main results reflect variation-dependent behavior rather than general unreliability.

2. Identifiability of the hidden rules

For program induction and symbolic Raven's matrices, what guarantees that the support examples uniquely determine the correct query outputs? Have the authors checked whether multiple rules or programs consistent with the support examples can produce different query predictions? If such ambiguous instances exist, how frequently do they occur, and how does removing them affect the reported results?

3. Statistical reliability of the reported gaps

How many independent base tasks are evaluated for each task family and transformation type? Could the authors provide confidence intervals for the main systematicity metrics and their differences, accounting for the grouping of variations under the same base task? This information is important for determining the stability of the conclusions, especially given the small number of variations and repeated attempts per task.

3 comments

Thank you for your positive assessment of our paper and the insightful and actionable feedback. We have conducted several additional experiments/analyses according to your suggestions that we believe have improved the paper.

Weakness 1. The central systematicity metric does not sufficiently distinguish transformation sensitivity from ordinary task-level unreliability.

Question 1. Separating stochastic unreliability from transformation sensitivity

Thank you for the concrete suggestion. We agree that it is important to distinguish transformation sensitivity from task-level unreliability to contextualize what is driving the observed systematicity gaps. Like you acknowledged, we attempt to do so in Section 5.2 with the variance decomposition and greedy decoding analysis and we extend both following your and Reviewer H6n4’s suggestions in the following:

  1. We now additionally perform a statistical test against your suggested null model to determine when the between task variation is statistically significant. Concretely, to test the null hypothesis that all variations of a base task share the same success probability, we compute a stratified Pearson's chi-squared homogeneity statistic across task variations and use a permutation test that shuffles attempt outcomes across variations of each base task to obtain p-values. The results are in line with the between task variation variances we reported (we updated Table 1 in the PDF).
  2. We extend the greedy decoding experiments beyond Gemini models. Since neither GPT nor Claude reasoning models allow greedy decoding, we do so on current, open-weight reasoning models Deepseek 4.1 Flash, GLM-5.3 and Kimi K3.1 Pro. Please kindly take a look at our general response to see the detailed results. Generally speaking, we observe similar gaps in systematicity in line with the findings on Gemini.

Like you mention in your response and we detail in the discussion, reliability is a prerequisite for strict systematicity and the strong within-task-variation variance plays a central role for the observed systematicity gaps. The greedy decoding analysis demonstrates that this lack of systematicity goes beyond ordinary stochastic sampling noise. We will revise the manuscript to make this distinction more prominently.

Weakness 2. The validity of the rule-induction tasks as unambiguous tests of systematicity is not sufficiently established.

Question 2. Identifiability of the hidden rules

Thank you for raising this important point. Rule induction tasks are ambiguous in nature and it is in principle possible that there exist multiple valid rules of a similar complexity to explain a given support set that lead to conflicting predictions on the query set. In our evaluations, we carefully control this ambiguity. In the Raven task family, it is computationally feasible to exhaustively enumerate the hypothesis space over all possible rule combinations and feature permutations. We verified that none of the tasks we evaluate have ambiguous answers.

In the program induction task the hypothesis space is infinite and we can only enumerate rules up to a certain complexity. This complexity is however considerably larger than the complexity of the ground truth rules. For the tasks we evaluate on, we verify that there exists no simpler or equally simple rule to explain a task. This is in large part due to partitioning the sequence lengths of the support and query sets such that the support set contains both shorter and longer sequences than contained in the query set. We will add this information to the manuscript.

Regarding your point on the distinction between formal structural equivalence and equivalent inference difficulty: We fully agree with this assessment and this is why when we generate task variations in Raven, we ensure that all variations are permuted, i.e. we do not compare permuted to unpermuted variations. The corresponding statement in the submission was not making this point sufficiently clear and we will revise it accordingly.

Question 3. Statistical reliability of the reported gaps

We have added error bars indicating the standard error of the mean to the main systematicity plot. We report the number of independent bases tasks per task family in the following table and will add this information to the manuscript. Note that we share a canonical task as the base task variation to obtain K=5 variations per transformation type.

Task familyBase tasksVariants per base taskAttemptsTotal tasks evaluated
Boolean98 (4 translation + 4 shuffle)5405
Raven104 (4 permutation)5250
Grammar1012 (4 translation + 4 recomposition + 4 shuffle)5650
Integer list128 (4 translation + 4 shuffle)5540

In terms of the confidence interval for the differences, we used the Wilcoxon signed-rank test to determine whether the systematicity gap is significant across tasks for each model.

ModelGap meanGap SEMWilcoxon p
Claude Opus 4.70.28750.06260.0004
GPT-5.50.16250.05000.0042
Gemini 3 Flash0.30420.06820.0004
Gemini 3.1 Flash-Lite0.19240.06370.0042
Gemini 3.1 Pro0.28750.06260.0004

Thank you again for your thoughtful review. As the discussion period is nearing its end, we wanted to kindly ask whether our response addresses your concerns? Please let us know if you have any additional questions.

Reviewer_znexOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper discusses the idea of models’ systematicity in their thinking and logic specifically using 4 benchmarks with rule induction tasks and show that basically models are not reliable in handling tasks even when the structure remains the same but the surface-level details that in theory should not affect a model that would have a specific type of reasoning, distract the models which basically puts models reasoning into question.

Critical strengths

I’m thinking probably the analysis that authors do in appendix C.1 is novel and the results which is basically the effect of reasoning on the sensitivity of the models are kinda interesting and I don’t know about that many works that have done such things. on top of that i think the study of the greedy decoding control and the isomorphic formalization that the authors offer is interesting as well.

Critical weaknesses

The idea is super interesting but I’m thinking probably others have already explored this idea ? Like for instance all the robustness or sensitivity analysis with analogical reasoning or other types of reasoning with llms. Either with math or analogical reasoning or others.

McCoy, R. T., Yao, S., Friedman, D., Hardy, M. D., & Griffiths, T. L. (2024). Embers of autoregression show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences, 121(41), e2322420121.

Wu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang, B., ... & Kim, Y. (2024, June). Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. In Proceedings of the 2024 conference of the north american chapter of the association for computational linguistics: Human language technologies (volume 1: Long papers) (pp. 1819-1862).

Lewis, M., & Mitchell, M. (2024). Evaluating the robustness of analogical reasoning in large language models. arXiv preprint arXiv:2411.14215.

authors are basically talking about the systematicity in a super similar framing compared to talking about the sensitivity and robustness analysis that people have done a lot so far. And this is basically my biggest concern about this paper, which its novelty and how its contribution is truly novel and it’s truly adding to the field

The examples that you’re mentioning in 049-051 are a bit too abstract and also from the current version, it’s not clear exactly what you mean, specifically in the second example, can you actually be more specific and use examples that are easier to understand specifically whether you mean semantic, structure, other types of generalizations, … ? Especially because right now it’s not clear whether you mean by learning A you have already learned A’ or that if you can learn A, you will also be able to learn A’ (your current second example right now reads as if the second statement is correct)

Are the last couple of citations like the ones from 128-130 already cover the contributions of this paper or is there a inherent specific thing that is different between this paper and the other papers that are either discussed here or are not discussed but are highly relevant ?

Is systematicity defined the way that you’re claiming it’s defined because I feel like you’re using a big definition but a small implementation of the idea; to reiterate, I mean you’re mostly talking about invariance to isomorphic surface transformations but not for instance other broader systematicity claims.

Overall the paper spends too much space and content just to explain systematicity and basically just to introduce the tasks although all these tasks have been discussed before either exactly with this form or just a small variation of them in the literature and the specific finding of the paper since it’s not really with any specific addition to them is not complete and enough to be publishable material.

Questions for authors

please check weaknesses.

8 comments

Thank you for your feedback on our paper, we are happy to hear that you find the idea interesting. We regret that our initial version of the submission did not make its novelty sufficiently clear and we will improve the manuscript accordingly. Please allow us to clarify how the present study differs from related work in the following.

Weakness 1: The idea is super interesting but I’m thinking probably others have already explored this idea ?

Please allow us to clarify the novelty of our work and explain why the mentioned references – while interesting and relevant – do not address the central question of our paper “Do reasoning models have systematicity of thought?”. For our study of systematicity of thought in reasoning models, the following elements are central:

  • Compositional task structure: Systematicity concerns structured changes that recombine familiar constituents and their structural roles (Fodor & Pylyshyn, 1988). We therefore need access to a data generating process with compositional structure to create controlled compositional task transformations.  

  • Inductive reasoning: We consider rule induction tasks which require inferring the compositional structure we seek to probe rather than supplying it, e.g., in a deductive task.

  • Reasoning models: Inference-time reasoning gives models an opportunity to "think" and as a result these models are significantly less sensitive to surface changes (Wei et al., 2022; Wang et al., 2022; Lightman et al., 2024; OpenAI et al., 2024). We would like to understand whether this thinking process gives rise to systematicity.

While measuring systematicity is a type of robustness analysis, the inverse is not true: A robustness analysis does not generally test systematicity of thought when task perturbations do not probe the reuse of constituents and operations across structured recombinations. The specific references you mention provide important foundations for our work but do not jointly examine these elements. McCoy et al. (2024) study sensitivity to task and example probability in non-reasoning models, rather than systematic transfer across compositional variations. Wu et al. (2024) test counterfactual task conventions in non-reasoning models but do not isolate the discovery and reuse of compositional structure. Lewis and Mitchell (2024) investigate variations of inductive analogical reasoning tasks that were unlikely to have been encountered during pretraining of non-reasoning models.

Weakness 2: The examples that you’re mentioning in 049-051 are a bit too abstract

Thank you for making us aware that these examples might be difficult to understand, we will revise the example in the manuscript. The pseudowords “blicket” and “wudsy” are often used in cognitive science literature and we use them as examples of novel concepts. For the second example in particular, your understanding is correct that we intend to say that “If you are able to infer A, you are also able to infer A’.”. This is meant to exemplify the “inductive reasoning” part of the rule induction tasks we study.

Weakness 3: Are the last couple of citations like the ones from 128-130 already cover the contributions of this paper

Continuing our response to Weakness 1, these papers study some form of robustness but do not test systematicity across structured recombinations of constituents and operations. Wang et al. (2023) assess trustworthiness, Zou et al. (2023) investigate adversarial attacks on alignment safeguards, Berglund et al. (2024) demonstrate a generalization failure when reversing factual associations, Mondorf et al. (2026) adversarially identify difficult problem instantiations that preserve the underlying logic.

Weakness 4: Is systematicity defined the way that you’re claiming it’s defined because I feel like you’re using a big definition but a small implementation of the idea

In Section 3.1, we state our definition of systematicity in the context of this study: “We define systematicity as invariance of ability with respect to certain transformations of a task with compositional structure. [...] . To quantify systematicity, we will measure how the ability to solve a task varies across task variations that modify the constituents of a task according to its compositional structure.” This is the reason why we carefully establish data generating procedures whose compositional structure we can use to test task variations that recombine the constituents of a task. For example, to permute task constituents in the grammar-based instruction learning task family or rebind features in the symbolic raven progressive matrices. The important point to note here is that the transformations we consider aren’t arbitrary isomorphisms but directly correspond to compositional task structure.

Weakness 5: Overall the paper spends too much space and content just to explain systematicity and basically just to introduce the tasks

Thank you for your feedback. We will revise the manuscript to more succinctly define systematicity and focus the description of the tasks on the compositional structure that we use to construct systematic task variations and use the freed space to move some key results that are currently shown in the appendix to the main text.

Thank you for the response. The clarifications on the examples in lines 049-051 and the plan to condense the systematicity and task descriptions address those points. However, my two main concerns remain.

Novelty relative to prior work. The response distinguishes this work by its combination of compositional task structure, inductive reasoning, and reasoning models. But the core experimental manipulation (holding the underlying structure fixed while changing the surface realization, and testing whether performance is preserved) is what the robustness and analogical-reasoning literature already does. Lewis & Mitchell (2024), for example, study inductive analogical tasks under exactly such structure-preserving variations. Whether or not prior work uses the term "systematicity," it tests the same thing. Applying this to reasoning models is an update in the models evaluated rather than a conceptual contribution. What does this setup reveal that a structure-preserving surface-invariance test could not?

Definition of systematicity. Pointing to the definition in Section 3.1 doesn't resolve this concern; it confirms it. The operational definition is much narrower than the systematicity of thought invoked in the title, abstract, and introduction. If the narrower notion is what the paper measures, why not frame the paper around it from the start?

Given this, I maintain my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all reviewers for taking the time to provide detailed feedback on our paper.

In response to Reviewer eKti and H6n4 we have run additional experiments in the greedy decoding setting. While the original results were limited to Gemini models since the other considered reasoning models (GPT Sol, Claude Opus) do not support greedy decoding, we now additionally evaluate three open-weight models which do support greedy decoding, finding qualitatively similar results outside the Gemini model family:

Task familyModelSolve anyFraction solvedSolve allΔ (any - all)
BooleanDeepSeek-V4.1-Flash-10171.0000.9330.8890.111
BooleanGLM-5.30.8890.7560.5560.333
BooleanKimi K3.1 Pro0.8890.8890.8890.000
GrammarDeepSeek-V4.1-Flash-10170.6670.4730.3000.367
GrammarGLM-5.30.0000.0000.0000.000
GrammarKimi K3.1 Pro0.8670.7070.5330.333
Integer listDeepSeek-V4.1-Flash-10170.7920.3750.0420.750
Integer listGLM-5.30.1250.0250.0000.125
Integer listKimi K3.1 Pro0.9170.8830.7920.125
RavenDeepSeek-V4.1-Flash-10170.8000.2800.1000.700
RavenGLM-5.30.2000.0800.0000.200
RavenKimi K3.1 Pro0.9000.5400.1000.800

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.