- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates what happens when reasoning in small language models is corrected at the first observed mistake in the trace. Reasoning is corrected using mainly two types of signals: localization (telling the model where the mistake happens) and providing the corrected state. The authors observe different responses from models of different sizes.
Critical strengths
Investigating how LLMs respond to corrective messages may, in the long term, inform the design of better training protocols for reasoning LLMs.
Critical weaknesses
- LLMs were overused in the writing of this paper, yielding text that was hard to follow in many critical parts.
- It is unclear if the findings observed generalize beyond the simple deterministic scenarios proposed.
- It is unclear what exactly do these synthetic problems look like.
- The findings may be over stated. They do not seem to be as consistent as stated.
Questions for authors
LLM usage was abused in this paper, yielding text that is unnecessary, confusing, and which considerably impacts the readability of the paper. Here are a few examples:
- "A matched message containing 32 can." How exactly does "32" help? This was not mentioned before and is not explained at all.
- I am not sure the mathematical symbol ⊭ is used correctly in lines 112 to 116. This symbol means “does not satisfy” or “does not semantically entail.” But it seems to be used in a positive sense in these lines (if I understand them correctly, which may not be true...). For example, the definition of Adopt should be that the correct state is adopted, not that the adoption property is not satisfied. In any case, the whole math in these lines is completely unnecessary, as it is never used again. The plain language definitions in Section 3.5 are much better.
- "The trade-off is reduced ecological breadth," - what does this mean in English?
- "The confirmatory set contains... " - This is the first time I hear of "confirmatory" sets. Is this dev or test? The difference matters.
- "Semantically related feedback forms were not interchangeable." - what does this mean?
- "The stable observation is the tension itself" - what does this mean?
The authors state that "We therefore study compact transition systems in which every intermediate state is known." But do the observed findings generalize beyond this simpler scenario? It is unclear. This limitation reduces the usefulness of the stated observations.
"Our controlled tasks make each transition deterministic." - are you sure the process is completely deterministic? Many mathematical operations are commutative, which produces many equally valid alternatives for a reasoning trace over a mathematical problem. The authors should include examples of problems and reasoning traces and discuss their deterministic nature more.
"Table 1 summarizes Qwen3 repair. Localization-only performance increases sharply with capacity, from near zero at 0.6B and 1.7B to 74.3% at 4B. Correct-state feedback has its largest effect at 1.7B: a 63.0-point paired gain. At 4B, however, localization alone outperforms structured correct feedback by 14.3 points." - Why is this happening? the "Correct-state" feedback should also improve with model size, no? In fact, it does for Gemma, just not for Qwen. Further, the "wrong state" feedback is also inconsistent for Qwen: its responses is largest at 4B for Arithmetic problems, but not for Logic, where it is much larger for the intermediate model (1.7B).