- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies whether language models distinguish dependency-preserving from dependency-violating reorderings of chain-of-thought steps. The authors use numerical variable references to construct step-dependency graphs and introduce a permutation probe that measures answer accuracy, confidence, and explicit judgments of ordering validity. Across six Qwen model variants, they report that some models are more sensitive to invalid permutations in their answer confidence, while explicit validity detection remains near chance. The paper then proposes Dependency-Aware Permutation Training (DAPT), which uses DPO to prefer valid over invalid orderings of the same steps and final answer. On Qwen2.5-1.5B-Instruct, DAPT increases mean GSM8K accuracy from 51.5% to 64.7% and step-ordering accuracy from 2.5% to 10.0% across three seeds. The authors also evaluate continuation, other benchmarks, and a 0.5B model, for which DAPT substantially reduces GSM8K accuracy.
Critical strengths
The paper asks a focused and worthwhile question: whether models distinguish dependency-preserving from dependency-violating step reorderings, rather than merely reacting to any change in step order.
Critical weaknesses
- DAPT (the proposed method) is modified from DPO, but only compared to SFT. This is unfair at all.
- In figure 1 (a), the arrow covers the step 2-3 blocks.
- Figure 2 and 3 looks like a draft. They are taking to much unnecessary space.
- The abstract is not clear enough to explain what is "structural dependency" and how the authors solve the proposed problem.
- There are multiple inconsistencies in the paper. The authors should check more for the paper before submission.
Questions for authors
- I think auto-research is an inevitable trend, but researchers should be responsible for their works when submitting them to any community, especially high quality conferences like ICLR. Could you please be more serious with your work. Thank you!
- DPO should be compared as a baseline method.