- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies an interesting and practical problem: if we rewrite a problem in an equivalent way (rotate a sudoku, swap two independent code lines, rename variables in a math question), the answer should not change, but diffusion LLMs often get it right in one form and wrong in another. They call this logical equivariance. They build a Sudoku benchmark where the targets are logically forced, so the answer transports exactly under rotations and reflections. They also evaluate code infilling and MATH-500 with paired transformations. Then they prove a theorem that row-major relative-score attention can only be exactly D4-equivariant if the offsets collapse into 2 or 3 classes. They propose a post-training recipe, RSM + PR, to get a better performance.
Critical strengths
-
The evaluation idea for dllm is good. Tracking each problem across all its views and reporting ANY/ALL makes clear something average accuracy hides. Eq. 4 is simple but convincing to me.
-
The Sudoku-AC-orbit construction is good. Keeping only states where the target cells are forced avoids the question of whether the answer really stays the same after the transformation.
Critical weaknesses
-
The data used to train RSM is not clearly decribed. I think the data has some changes between experiments. In 5.4, "RSM uses the 4.6k puzzles for which the deduction procedure produces a trajectory, yielding 41k intermediate states." and in appendix, "The cache contains 67,551 states from 9,995 of 10,000 puzzles". I didn't get the difference. And I also found the comparsion is not 100% fair, see next point.
-
RSM uses 4.6k trajectories, but canonical uses all. And also, the mask density is also different. It's good to have appednix C. But the contrast between A and C is a negative result if I understand it correctly, which means RSM doesn't work that well.
-
Since AR models are still the mainstream, I think it's important to mention or set some AR models as baseline. Prompt-order and format sensitivity is well known in AR models (for example this paper: Multidimensional Consistency Improves Reasoning in Language Models), so without this comparison I cannot see what is specific to diffusion.
Questions for authors
From the weakness, I would like to ask:
-
Why is the RSM training data different between Section 5.4 (4.6k puzzles, 41k states) and Appendix C (9,995 puzzles, 67,551 states)? Is the state generator different?
-
How do you explain the negative result in Appendix C?
-
Maybe some AR models could be included?
2 comments
Response to Reviewer ByqZ (2/2)
W3 / Q3. AR baselines: The phenomenon is shared.
We ran Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct with greedy decoding.
Sudoku orbits, 500 problems × 8 views:
| Model | Window | Mean success | ANY | ALL | Parse failures |
|---|---|---|---|---|---|
| Qwen2.5-7B | AC4 | 57.5 | 100.0 | 1.6 | 663/4,000 |
| Qwen2.5-7B | AC16 | 6.7 | 42.6 | 0.0 | not tallied |
| Llama-3.1-8B | AC2 | 27.4 | 87.0 | 0.0 | 25/4,000 |
| Llama-3.1-8B | AC4 | 13.2 | 63.8 | 0.0 | not tallied |
An AR model that solves 57.5% of views solves all eight views of a puzzle 1.6% of the time. Parsing does not explain this. For Llama AC2, at least 410/500 puzzles have all eight outputs parsed and still mix correct and incorrect answers. For Qwen AC4, at least 148/500 puzzles contain both a parsed-correct and a parsed-incorrect view.
MATH pairs, greedy (582 original–transformed pairs from 328 base problems; accuracy / ANY / ALL at the pair level, as in Table 2):
| Model | Accuracy | ANY | ALL | ANY − ALL |
|---|---|---|---|---|
| Qwen2.5-7B (AR) | 36.0 | 41.6 | 30.4 | 11.2 |
| Llama-3.1-8B (AR) | 31.7 | 39.0 | 24.4 | 14.6 |
| LLaDA-8B (diffusion, Table 2) | 29.0 | 38.7 | 19.4 | 19.3 |
Parsing failures (Qwen 25/910 inputs, Llama 10/910) are counted as incorrect. All three models change correctness under answer-preserving rewrites of the same problem.
What the AR baselines establish.
- Presentation dependence is not unique to diffusion. Prompt sensitivity in AR models is well known (e.g., Multidimensional Consistency Improves Reasoning in Language Models, now cited). What was missing is an exact test with targets that provably transport across presentations, ensuring disagreements are true errors rather than ambiguities. Our construction supplies this test, and it applies directly to AR models.
- Flexible generation order is not a solution. Arbitrary-order generation might seem like a natural defense against layout variations. Table 2 shows it is not. Generation order and input presentation are different axes.
- Masked diffusion enables our controlled intervention. Its explicit training distribution over partial states allows us to vary practiced conditional tasks while strictly matching data, mask density, and per-target supervision. Under this control at 100M, the identical PR augmentation lowers mean success on random states (−1.00) but raises it on reasoning states (+1.44).
- Architectural scope. The rigidity theorem assumes full, unmasked attention. This matches the masked-diffusion architecture, which is where we claim and test the structural result.
Response to Reviewer ByqZ (1/2)
We thank the reviewer for the careful reading and for the positive view of the evaluation idea and the Sudoku-AC-Orbit construction. We address each weakness and question below with exact numbers and new experiments.
(Table numbers in this response refer to the original submission. In the revised PDF: original Table 2 → Table 1; Table 4 and Table 8 → Table 11; Table 15 → Table 23; Tables 17–18 → Tables 26–27. New results: decoding controls in Table 7 and Table 24, three-seed recipes in Tables 3 and 14–15, matched factorial in Table 28, token calibration in Table 29, refresh intervention in Table 30, matched mazes in Table 22, AR baselines in Tables 8–9, inference baselines in Table 10.)
W1 / Q1. One generator, two retention rules.
Five of the 10,000 puzzles emit no state. We'll added this table to the revision to clarify the data split.
W2. The comparison the review calls unfair is not the one our claim rests on. Appendix C exists to remove exactly the confounds you list, and we have now closed the last one.
With all three matched, the RSM × PR interaction in mean success is +1.51 [1.04, 1.98] at 50M (5/5 seeds) and +2.45 [1.89, 3.01] at 100M (3/3 seeds). The interaction in sECR is +4.06 [2.61, 5.56] at 50M and +6.32 [4.70, 7.92] at 100M. Data amount, mask density, and target allocation do not explain this effect.
Q2. A vs C is not evidence that RSM fails. It is half of the result.
A vs C compares the two state types without PR. Reasoning states lower sECR by 4.36 points at 50M and 6.57 [5.18, 7.95] at 100M under matched weights. The other half is what happens with PR.
We also explain why A vs C is negative, and test the explanation.
Together, these results support presentation persistence as a mechanism for the state-dependent PR effect. The positive matched endpoints separately establish where the augmented reasoning-state model solves more.