- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates symbolic reasoning failures in LLMs through regular expression interpretation and DFA construction. The authors identify recurring failure modes, evaluate fine-tuning strategies, and conduct mechanistic analyses to understand why difficult cases remain unresolved. The results suggest that some failures persist despite training and inference-time interventions.
Critical strengths
- The paper studies an important question concerning the mechanisms underlying symbolic reasoning failures.
- The evaluation uses formally verifiable tasks, providing reliable correctness criteria.
- The study combines behavioral evaluations, fine-tuning experiments, and mechanistic analyses in a systematic framework.
Critical weaknesses
1.The mechanistic analysis relies heavily on correlations and averaged activation patching. These experiments do not convincingly establish that the observed failures originate from fundamental limitations in the model's internal representations, weakening the paper's central claim.
2.The curriculum experiments differ from joint training in more than training order, while dataset deduplication does not guarantee language-level uniqueness. These issues weaken the conclusions about generalization and the limits of training-based mitigation.
Questions for authors
- Can the authors provide stronger causal evidence, such as instance-matched activation patching, to support the mechanistic conclusions?
- Can the curriculum experiments be repeated with identical data composition and supervision formats?
- Have the authors checked for semantically equivalent regular expressions across training and evaluation sets?
10 comments
Response to Weakness 1 and Question 1 (Continued)
Our mechanistic conclusions do not rest on correlations. The paper's correlational analyses, the neuron scores and the linear probes, are used mainly to find where to look. Each conclusion is then tested by changing the model's internals and measuring the effect on its answers, at three levels: single neurons, directions in activation space, and the full internal state of each layer. Across all of them, including the new instance-matched test, no intervention repairs more than a small share of the failures:
- Neurons (Table 2c): amplifying or damping the top-scoring neurons does no better than doing the same to random neurons. In a per-problem rerun, it changes the outcome of at most 3 of the 285 problems.
- Directions (Table 2c): steering along learned directions (ActAdd, the probe direction, SADI) gains at most +1.0 point, and none of these changes is significant.
- Iterative steering (Table 2c): the strongest intervention, VGNS, fixes 7 of the 42 failures (paired p = 0.016), and 35 resist all four rounds.
- Full state, averaged (Section 5.2): replacing the state with the average correct state fixes none of the 42 failures at layers 0 to 25 or with all 28 layers at once. The few changes at the last two layers also appear when the average failing state is patched in instead, so they reflect disturbance rather than repair. Because an average also removes the information about which regex is being solved, the instance-matched test below is the stronger version of this experiment.
- Full state, instance-matched (new): replacing it with a matched twin's correct state fixes none of them, at any of the 22 layer settings.
Across every level we examined, no intervention repairs more than 7 of the 42 failures. The Layer-10 state carries a moderate signal of the outcome (AUROC 0.85, precision 0.54), but the failure resists every local correction, including the correct state of a matched problem, which is what the paper means by a structural failure. As in the paper, we state this conclusion for the interventions tested. We will add the instance-matched experiment to Section 5.2 and Appendix F, and replace the sentence at lines 452–454 accordingly.
Response to Weakness 2a and Question 2
We thank the reviewer for this point. The reviewer is right that our original curriculum runs differed from joint training in more than order: they added theory Q&A examples, a balanced replay phase, a different trace format, and different epochs, batch size and early stopping (Table 5, lines 1300–1310). We have rerun all five orders with these differences removed. The reviewer's related question on duplicate languages is answered in W2b and Q3 below.
Curriculum with order as the only difference. Each run uses exactly the joint-training data (the same 22,500 construction examples, with no theory Q&A and no replay), the same JSON-trace supervision, and the same LoRA configuration, optimizer, epochs, batch size and seed as joint training. The learning rate follows a single cosine schedule over the full run, as in joint training. The examples are grouped by tier and presented in the stated order. As a check on the training code, the same script with all tiers shuffled together reproduces joint training: 96.5% overall and 82.9% on Tier 4, the same as in Table 2a. All runs are evaluated on the same 200-example test sample as Table 2b (39, 47, 73 and 41 examples in Tiers 1–4, where Tier 1 includes the sample's one handcrafted example).
| Order (7B, CoT) | Tier 1 | Tier 2 | Tier 3 | Tier 4 | Overall | Original Table 2b |
|---|---|---|---|---|---|---|
| Natural (1→2→3→4) | 56.4 | 66.0 | 56.2 | 46.3 | 56.5 | 43.5 |
| Reverse (4→3→2→1) | 48.7 | 40.4 | 24.7 | 19.5 | 32.0 | 26.0 |
| Hard-first (4→1→2→3) | 53.8 | 44.7 | 38.4 | 31.7 | 41.5 | 39.0 |
| Mid-out (3→2→1→4) | 61.5 | 53.2 | 34.2 | 48.8 | 47.0 | 43.5 |
| Random permutation (1→4→2→3) | 59.0 | 51.1 | 39.7 | 56.1 | 49.5 | 37.0 |
| Joint (all tiers shuffled) | 100 | 100 | 100 | 82.9 | 96.5 | 96.5 (Table 2a) |
The collapse is caused by ordering, not by the extra components. With order as the only difference, the best curriculum reaches 56.5% overall against 96.5% for joint training. The best Tier 4 result under any order is 56.1%, 26.8 points below joint training's 82.9%. The extra components of the original runs did make results worse, lowering overall accuracy by a further 2.5 to 13 points depending on the order. But those components only enlarged the collapse. They did not cause it. So changing the order of training does not close the Tier 4 gap, which addresses the reviewer's concern about the limits of training-based mitigation.
What carries over from Table 2b, and what changes. Easy-to-hard (natural) order is still best overall, and the orders that begin with the hardest tier are still worst (reverse 32.0%, hard-first 41.5%). The middle of the ranking changes: the random permutation rises from fourth to second, and it gives the best Tier 4 accuracy (56.1%). With only 41 Tier 4 test examples (95% intervals of about ±15 points), the smaller Tier 4 differences among orders, such as random (56.1%) against natural (46.3%), are within sampling error. The gap to joint training is not.
Interpretation. Under every order, tiers trained early fall far below their near-perfect joint-training accuracy, which is consistent with forgetting. Tiers trained last also stay low, for example Tier 4 at 46.3% under natural order and Tier 1 at 48.7% under reverse order. Because the learning rate has largely decayed by the final phase of the single cosine schedule, this second effect may reflect reduced plasticity, a low final learning rate, or both. We will report these controlled runs in Table 2b and Appendix C.
Response to Weakness 1 and Question 1
We thank the reviewer for the careful review and for recognising the importance of the question, the use of formally verifiable tasks with exact correctness criteria, and the systematic combination of behavioural evaluation, fine-tuning, and mechanistic analysis. The reviewer asked for three things: instance-matched activation patching, a curriculum experiment with identical data and supervision format, and a check for semantically equivalent regexes across training and test. We ran all three, reported below.
W1 + Q1.
We thank the reviewer for this request. We agree that averaged patching is a weak test: it replaces the representation of the specific regex with an average over different problems, so it cannot show whether an instance-correct state would repair a failure (lines 445–447). We have now run the stronger test the reviewer asks for. On each failing problem, we transplant the internal state of a matched twin that the model solves correctly. No transplant repairs a failure, at any of the 22 layer settings we tested.
Why this test is now possible. The submission noted that its alphabet-swapped variants could not support this test, because none of them was both solved correctly and token-aligned with its failing base (lines 452–454). For this rebuttal, we searched much more widely for twins: we renamed symbols both within the original alphabet and to new symbols, and checked every candidate, one problem at a time, for both a correct solution and exact token alignment.
How twins are built. A twin of a failing Tier 4 problem is the same problem with its symbols renamed. Only the regex symbols and the alphabet list change. The automaton has the same structure, and the rest of the prompt is identical. We keep a twin only if:
For example, the model fails (0|3)((02+1+++))*2 but solves its twin (2|0)((21+3+++))*1. The two 140-token prompts differ at only 6 positions. We found 14 verified pairs, covering 10 of the 42 failures. These 10 are the failures closest to success, since the model solves the same problem under a different symbol assignment. If any failure could be repaired by supplying a correct state, it should be one of these. For the remaining 32 failures, our search found no twin that the model solves and that is also token-aligned.
What we transplant. We run the failing prompt, but at the chosen layers we replace its activations with the correct twin's, then let the model generate. We test 22 layer settings for every pair: 15 single layers (every second layer from 0 to 26, and 27), six consecutive blocks of 4 to 5 layers (0–4, 5–9, 10–14, 15–19, 20–23, 24–27), and all 28 layers at once. We use two position choices:
Results (CoT 7B, Tier 4). In every row, the model works on the problem it fails. Each row says whose internal state we put into that run.
The two columns check two different outcomes. The first counts how often the failing problem is now solved correctly. The second counts how often the model instead builds the twin's correct DFA, which would mean it simply copied the twin's answer. Both are zero everywhere.
As a check, we also copied each failing run's own state back into itself. Its answer stayed exactly the same, which confirms that the copying step does not change the model's behaviour on its own.
The main result is that a correct twin's state never fixes a failure, at any of the 22 layer settings, including all 28 layers at once. It does not even make the model copy the twin's answer. So the failure does not come from a faulty internal state that a correct one could replace, whether at a single layer, a block of layers, or all layers at once.