- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces a framework for investigating how training LLMs on valid versus invalid reasoning steps affects their behavior. Across four LLMs, training with valid reasoning steps improves a validity margin. The authors analyze how representational differences, activation-patching effects, and parameter-replacement effects vary across network depth. They demonstrate that these quantities exhibit different, model-dependent trends and do not necessarily coincide.
Critical strengths
Investigating the consequences of training on invalid logical steps is an interesting and worthwhile research direction.
The matched training setup allows for a relatively controlled comparison of the effects of valid versus invalid reasoning supervision.
The paper identifies rather different depth-wise trends across three small language models, which is an interesting empirical observation.
Critical weaknesses
The primary problem of this paper is the writing, which makes the main contribution hard to evaluate. Throughout, many sentences are unnecessarily complicated and verbose, and many new terms are introduced (e.g. causal footprint, causal maturation, causal expression, finite causal leverage, directional refinement, prediction-associated representations, validity-related direction, parameter replaceability, causal geometry, and many more), some of which are hard to understand. The abstract is not very helpful to the potential reader as it is overly technical and uses terminology that is introduced by the paper itself (e.g. reasoning-sensitive Terminal validity margin, Bidirectional activation transplantation). The same is the case for the introduction. Some mathematical quantities are not properly defined, e.g.
The main behavioral metric, terminal validity margin, measures likelihood preferences over given candidate continuations rather than letting the model produce a reasoning trajectory itself, limiting the conclusions about reasoning capabilities and making the higher terminal validity margin for the healthy training is unsurprising as a direct artifact of the training setup.
The substantially different depth-wise trends across models could have been discussed more.
Questions for authors
Have you considered evaluating generated reasoning on independent, out-of-distribution tasks to test robustness of measured behavioral differences?
Can you provide possible explanations for the substantially different depth-wise trends observed across models?
2 comments
Questions and Additional Findings
Q1. Generation on independent, out-of-distribution tasks
We added generation on the official ProofWriter test split: 1,200 provable queries from 600 sets of facts and rules, covering depths one to three.
This extends the original ProofWriter candidate evaluation to generated answers and supporting facts and rules. This tests whether models generate correct responses with sufficient evidence.
Paired two-/three-step problems separately test ordered proof generation and stopping.
Q2. Why do depth-wise trends differ across models?
Three factors matter: the size of the change to an internal state, its direction, and how strongly subsequent computation responds to it. A large change can have little effect if it points away from directions that change the score.
In the original notation, measures normalized state difference. is alignment with the score gradient, the direction of steepest local increase. averages the score gain from inserting H states into S and the loss from inserting S states into H.
The layer pairs below were selected on discovery data and evaluated on held-out questions:
| Model and layer pair | Change in | Change in (nats/token) | What changes? |
|---|---|---|---|
| Qwen2.5-3B, L21→L27 | +0.01582 | +0.01476 | Better alignment accompanies a stronger effect. |
| Ministral-3B, L15→L19 | +0.00651 | +0.01695 | The effect strengthens despite a smaller normalized state difference. |
| LLaMA-3.2-1B, L10→L15 | −0.01033 | −0.00550 | Alignment and effect weaken after an earlier peak. |
Both contrasts have these signs in all three seeds per model (Appendix E, Tables 14–16).
These profiles reflect model-specific combinations of alignment and sensitivity. Architecture and pretraining are plausible contributors, whose separate effects would need to be tested in controlled comparisons.
Additional findings: effects on proof completion
Using separate Qwen2.5-3B checkpoints with matched warrant-text frequencies, we evaluate 600 paired problems across three training seeds. Each pair has the same starting facts and target conclusion but requires two or three rule applications. A complete proof must apply the rules in a valid sequence and reach the correct answer. All % changes below are absolute differences in rates.
1. Better rule selection does not guarantee a complete proof. Given the correct starting fact, the H–S differences are:
| Generated behavior | Two-step problems | Three-step problems |
|---|---|---|
| Correct first rule | +2.89% | +4.33% |
| Correct answer and complete valid proof | +9.17% | −7.33% |
For the stopping test, we supplied the correct first two steps. Requiring one more rule choice reduced the three-step H–S third-rule gap from −22.56% to −0.50%, and the complete-proof gap from −18.56% to −5.17%. The model chose among six rules; the correct choice was not supplied.
2. Changing internal states changes stopping and proof completion. After the same two-step prefix, we intervened once at L27 on the last prompt token, moving H states toward S, then let the model generate. Premature answering fell by 10.33%, and three-step complete-proof accuracy rose by 5.17% (95% CI [+4.00%, +6.44%]). On two-step problems, where the proof was already complete, the same intervention reduced complete-proof accuracy by 7.50%. Its benefit therefore depends on whether another inference is needed.
3. Alignment explains the stronger L27 stopping effect. We define a stopping score as the log-probability of starting the answer minus that of continuing with a rule. For a single state change of norm , the standard first-order relation is
=aG_{\mathrm{stop},\ell}C_{\mathrm{stop},\ell}+\epsilon_\ell.$$ Here $G$ is the gradient norm, measuring local sensitivity; $C$ is cosine alignment with that gradient; $\epsilon$ is the difference between the measured effect and this approximation. The original $R$ is normalized state difference, not $a$. With S as recipient and the same norm at both layers, the L27–L21 stopping-score effect is **+0.606 nats**. Decomposing each problem before averaging gives **+1.000 from alignment, −0.364 from sensitivity, and −0.030 from the remainder**. The larger effect comes from better alignment despite lower sensitivity. This single-recipient score change is distinct from the original bidirectional $F$. The interval resamples paired problem groups, keeping the three training seeds fixed. **These experiments connect the original internal analysis to a concrete generation decision: whether to continue a proof or answer. Better initial rule selection can be offset by answering too early.**
Responses to Weaknesses
Thank you for the review. We apologize that some terminology was insufficiently explained. Below we clarify the definitions and their relation to prior work, and present new generation and intervention results.
W1. Writing, definitions, and terminology in prior work
A warrant is the rule licensing an inference. Healthy (H) uses a valid warrant; Severe (S) uses a corrupted warrant that does not justify the target inference. Training examples are matched by their original problem and target inference.
Our contribution is an empirical investigation of three properties of this supervision contrast: differences in internal states, effects of swapping states, and gains from swapping weights. These measurements need not peak at the same layers.
We distinguish established methodological terms from the labels used to describe our experiments:
These references use the related methodological terms; the DWL-specific definitions are given above. We will define each term and symbol at first use and simplify the abstract and captions.
W2. Candidate scoring versus generated reasoning
The original margin provides a common measure for comparing internal interventions. Generation tests measure how the supervision contrast affects model-generated answers and proofs.
On the official ProofWriter open-world test split, models generate an answer and supporting fact/rule IDs. For 1,200 provable questions, the checker requires a correct answer and verifies that the cited support derives the claim or its negation, as appropriate. No candidate continuation or gold proof is supplied for the test question.
This checks support sufficiency; the separate paired experiments in Part II check each step of an ordered proof.
Training was expanded to 84,111 instances per condition, approximately 12.6M loss-bearing tokens for Qwen2.5-3B and 13.2M for Qwen3.5. Models with positive mean H–S differences are:
Here % denotes absolute differences between success rates. Means include all completed seeds per listed model; 4B uses one seed. These results extend the supervision contrast to generated answers with verified support.
W3. Model-dependent depth trends
The differing depth profiles are part of the finding. Part II shows how the same supervision contrast produces different patterns of representation change, alignment, and causal influence across models.