- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether reasoning models perform consistently across structurally equivalent tasks. It focuses on structure-preserving task isomorphisms (symbol substitution, feature rebinding, constituent permutation and example reordering). Four families of problems are considered: grammar-based instruction-learning, symbolic Raven matrices, program induction over integer sequences and Boolean concept learning. For each task, the authors generate 5 variants per type of invariance. A variant counts as solved if at least 3 of 5 attempts answer correctly the entire query set. Results are reported for Gemini 3.1 (Flash-Lite, Flash and Pro), Claude Opus 4.7 and GPT-5.5, using 3 metrics: whether any variant is solved, the fraction solved, and whether all are solved, averaged over variation sets. Three task families show substantial gaps, which the authors interpret as a lack of systematicity. Further experiments analyze sampling noise, greedy decoding, adding hints, changing task difficulty and varying reasoning effort.
Critical strengths
-
The problem and task setup are clear. Synthetic variant generation through isomorphisms makes it explicit what differs between variants, and the approach is general enough to be applied to other tasks.
-
The distinction between within-task-variation noise (model stochasticity) and between-task-variation variance (task variations) is important. The variance decomposition analysis helps separating estimated differences between variants from stochastic noise across attempts. This helps explaining what contributes to the observed gap.
Critical weaknesses
-
The gap between solving at least one variant and solving all variants can arise from both stochastic unreliability and changes in performance when the task is reformulated. In Appendix A these sources of variance are usefully separated, with little estimated between-variant variance in several conditions. The authors also state that systematicity requires reliability of performance. The paper should make clear in the main discussion where the evidence supports sensitivity to task transformation and where it reflects unreliable performance. The zero-temperature Gemini experiment provides additional evidence of sensitivity to task reformulation, but does not quantify the influence of noise under default sampling.
-
Even if only isomorphisms are considered, some variants may not be equally difficult, but the analysis still assumes they are. In Section 4.2, the paper reports the permuted Raven example as "significantly more difficult because the features are no longer spatially aligned across columns". The Raven task family also has some of the largest between-variant variance estimates (Table 1). This makes it difficult to interpret the gap as a comparison between equally difficult tasks, even with the transformations being valid isomorphisms.
More generally, the task generators and controlled transformations are useful, but the potential analysis of the systematicity remains still limited. In the related work it is acknowledged that sensitivity to task reformulations is already well studied. The paper needs to explain better what effects persist beyond sampling noise and difficulty differences, to understand how these rule induction tasks aid in the understanding of systematicity.
Minor point: in the example in Section 4.3 the query has length equal to one sequence in the support set, which is in contradiction with what is written in the "Support and query set" paragraph. This may concern only the example.
Questions for authors
- If each variant of a task has the same success rate (per-attempt), what gap is expected and how does it compare with the observed gap once one accounts for the uncertainty?
- At zero sampling temperature, do repeated attempts of the same task variant give the same answer with Gemini?
- Regarding the Raven's problem, what happens if the analysis separates between aligned and misaligned variants? Do differences between variants persist when alignment is consistent?
- How many distinct base tasks and variation sets are considered per family? Were multiple variation sets generated from same base task? These should be reported in the main text.
- Please report confidence intervals accounting for variants from same base task.
- With structured output, do failures due to formatting occur? If so, how often do they occur and how are they scored?
For future work: an analysis of the reasoning traces of failed variants could be helpful.
6 comments
Question 4: How many distinct base tasks and variation sets are considered per family? Were multiple variation sets generated from same base task? These should be reported in the main text.
We report the number of independent bases tasks per task family in the following table and will add this information to the manuscript. Note that we share a canonical task as the base task variation to obtain K=5 variations per transformation type.
| Task family | Base tasks | Variants per base task | Attempts | Total tasks evaluated |
|---|---|---|---|---|
| Boolean | 9 | 8 (4 translation + 4 shuffle) | 5 | 405 |
| Raven | 10 | 4 (4 permutation) | 5 | 250 |
| Grammar | 10 | 12 (4 translation + 4 recomposition + 4 shuffle) | 5 | 650 |
| Integer list | 12 | 8 (4 translation + 4 shuffle) | 5 | 540 |
Question 5: Please report confidence intervals accounting for variants from same base task.
We added error bars reporting the standard error of the mean to the systematicity bullet plot. In terms of the confidence interval for the differences, we used the Wilcoxon signed-rank test to determine whether the systematicity gap is significant across tasks for each model.
| Model | Gap mean | Gap SEM | Wilcoxon p |
|---|---|---|---|
| Claude Opus 4.7 | 0.2875 | 0.0626 | 0.0004 |
| GPT-5.5 | 0.1625 | 0.0500 | 0.0042 |
| Gemini 3 Flash | 0.3042 | 0.0682 | 0.0004 |
| Gemini 3.1 Flash-Lite | 0.1924 | 0.0637 | 0.0042 |
| Gemini 3.1 Pro | 0.2875 | 0.0626 | 0.0004 |
Question 6: With structured output, do failures due to formatting occur? If so, how often do they occur and how are they scored?
Formatting failures occur rarely with the reasoning models considered here and when they do occur we retry until we get a valid response.
Thank you again for your constructive feedback that we believe helped improve our work. With the end of the discussion period coming closer, we wanted to kindly ask whether our response addresses your concerns? Please let us know if you have any further questions.
Thank you for taking the time to carefully review our paper. We have run several additional experiments and analyses following your suggestions that we think have improved the paper. We address your specific comments point-by-point below.
(Please allow us to copy our response to Reviewer eKti who raised a very similar point)
We agree that it is important to distinguish transformation sensitivity from task-level unreliability to contextualize what is driving the observed systematicity gaps. We attempt to do so in Section 5.2 with the variance decomposition and greedy decoding analysis and we extend both following your and Reviewer eKti’s suggestions in the following:
We now additionally perform a statistical test against a null model to determine when the between task variation is statistically significant. Concretely, to test the null hypothesis that all variations of a base task share the same success probability, we compute a stratified Pearson's chi-squared homogeneity statistic across task variations and use a permutation test that shuffles attempt outcomes across variations of each base task to obtain p-values. The results are in line with the between task variation variances we reported (we updated Table 1 in the PDF).
We extend the greedy decoding experiments beyond Gemini models. Since neither GPT nor Claude reasoning models allow greedy decoding, we do so on current, open-weight reasoning models, Deepseek 4.1 Flash, GLM-5.3 and Kimi K3.1 Pro. Please kindly take a look at our general response to see the detailed results. Generally speaking, we observe similar gaps in systematicity in line with the findings on Gemini.
As we detail in the discussion, reliability is a prerequisite for strict systematicity and the strong within-task-variation variance plays a central role for the observed systematicity gaps. The greedy decoding analysis demonstrates that this lack of systematicity goes beyond ordinary stochastic sampling noise. We will revise the manuscript to make this distinction more prominently.
We fully agree with you that structural equivalence formalized through isomorphisms does not by itself ensure equivalent inference difficulty. We carefully choose our task variations so as not to change their task difficulty. For the task variations on Raven you mention specifically, we ensure that all variations are permuted variants, i.e. we do not compare unpermuted to permuted instances. The corresponding statement in the submission was not making this point sufficiently clear and we will revise it accordingly.
We appreciate the suggestion and will revise the related work to more clearly emphasize the difference to prior work that evaluates robustness. Systematicity concerns structured changes that recombine familiar constituents and their structural roles (Fodor & Pylyshyn, 1988). We therefore consider data generating processes with compositional structure that allow us to create controlled compositional task transformations. This means that measuring systematicity is a type of robustness analysis but the inverse is not true: Perturbations by prior work do not systematically probe the reuse of constituents and operations across structured recombinations.
Thank you for pointing out this inconsistency in our example in Section 4.3, we will fix it in the paper.
We tested this for a subset of tasks and found the answer to be stable across attempts.
When we generate task variations in Raven, we ensure that all variations are permuted, i.e. we do not compare permuted to unpermuted variations. The corresponding statement in the submission was not making this point sufficiently clear and we will revise it accordingly.