- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper examines LLaDA-8B-Instruct on tasks with an explicit pointer structure. A symbolic parser identifies relevant records, which are presented sequentially or all at once. LLaDA is trained with LoRA adapters on single records, a path-set, and all records. The single-record LLaDA achieves 100% accuracy up to depth 12 chains, when trained at depth 4 and 8, versus 62.5% for the full-context input variant. Subsequent analysis finds that the full context / path-set fine-tuned LLaDA confuses repeated digit values. The proposed contribution is primarily diagnostic and attributes the gains to the additional structure.
Critical strengths
- The finding that static readers make errors when confusing different occurrences of the same digit suggests that the reader tracks its position through its output value rather than the nodes, which might have useful consequences, although the paper does not substantiate this explanation.
- The paper is transparent in its setup and the attribution of results.
Critical weaknesses
- The contribution is limited and not of clear benefit to the established literature. It is unclear what substantial insight or general understanding the paper establishes. The method uses an 8B dLLM to extract the value of records, but a piece of code could perform the same task. The conceptual contribution is not clear from the discussion; decomposing a problem into easier subproblems is already an established method. The related work does not sufficiently engage with other analyses of such mechanisms. For an empirical contribution, the paper considers one model in one specific setting. Both are insufficient to support an acceptance decision.
- The paper does not provide a general explanation. The considered chains are solved perfectly by the parser itself, and the LoRA fine-tuning does not directly relate to a realistic reasoning setting. Under replay, the parser already solves the chain itself; "extrapolation from 8 to 12 without error" is thus trivial.
- The stated "Loses its place" failure mode could be by construction; the nodes have no position index, and the static reader solves exactly that problem; it adds the idea of position back to the task.
- The headline 37.5 point-gap is measured with the full-context reader on path sets, which forms a train/test mismatch. The static reader was unstable over seeds, with constant LR, equal optimizer steps for the static reader could under train it, thus not overfitting. Tuning the reader would solve that.
- Nothing in the paper is diffusion-specific.
Questions for authors
- Could the authors emphasize what the principal scientific claim is? Is it a state-tracking mechanism or an evaluation methodology, and what experiment distinguishes it from prior work?
- Are the insights reproduced in other models and more realistic settings?
- An AR model could also receive one record at a time, or we rebuild its context, why do we solely consider a diffusion language model?
- What part of the reported gap is attributable to the training recipe?
- Could the authors give context to weakness (3)?
- Could the authors tune the static reader as well, to deal with the possible mismatch noted in (4)?
- Why is the paper focused on dLLM's, when the methodology does not appear to be specific to that class of models?