acceptodds
Withdrawn from ICLR 2027

Keeping the Reader's Place: Why Revealing Evidence One Step at a Time Helps Masked Diffusion Language

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A masked diffusion language model rereads its prompt on every forward pass, so what the prompt shows can change while the answer is being written. Revealing one record at a time along a chain of pointers turns a hard multi-hop prompt into a sequence of easy local reads, and it is tempting to credit the gain to a model that has learned where to look. We take the gain apart on tasks whose dependency structure is explicit, holding the model, the items, and the forward passes fixed while varying what is visible when, what the reader was trained on, and whether the order of reads is decided during generation or before it. A LLaDA-8B model fine-tuned to read one record at a time solves chains longer than any it saw in training. The same model fine-tuned on full prompts and handed exactly the same records all at once falls well short, and its errors have a consistent signature: when a value recurs in the chain, it resumes from the wrong occurrence, far more often than chance predicts. Revealing one record at a time removes this bookkeeping from the reader. The order that does so need not come from the model: a text parser computes it from the prompt before generation begins. Two further controls locate what remains. The strength of a static baseline depends on what it was trained on, and when neighboring answers are independent, as on chains of IMDB reviews, classifying each record on its own does at least as well as reading the chain on average. Revealing evidence one step at a time helps a diffusion reader because it keeps the reader's place, and the schedule that does so can be fixed in advance.

Primary area:
generative models
Keywords:
Generative Models, Reasoning, Natural Language Processing, Evaluation & Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_Eb3XOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper examines LLaDA-8B-Instruct on tasks with an explicit pointer structure. A symbolic parser identifies relevant records, which are presented sequentially or all at once. LLaDA is trained with LoRA adapters on single records, a path-set, and all records. The single-record LLaDA achieves 100% accuracy up to depth 12 chains, when trained at depth 4 and 8, versus 62.5% for the full-context input variant. Subsequent analysis finds that the full context / path-set fine-tuned LLaDA confuses repeated digit values. The proposed contribution is primarily diagnostic and attributes the gains to the additional structure.

Critical strengths
  1. The finding that static readers make errors when confusing different occurrences of the same digit suggests that the reader tracks its position through its output value rather than the nodes, which might have useful consequences, although the paper does not substantiate this explanation.
  2. The paper is transparent in its setup and the attribution of results.
Critical weaknesses
  1. The contribution is limited and not of clear benefit to the established literature. It is unclear what substantial insight or general understanding the paper establishes. The method uses an 8B dLLM to extract the value of records, but a piece of code could perform the same task. The conceptual contribution is not clear from the discussion; decomposing a problem into easier subproblems is already an established method. The related work does not sufficiently engage with other analyses of such mechanisms. For an empirical contribution, the paper considers one model in one specific setting. Both are insufficient to support an acceptance decision.
  2. The paper does not provide a general explanation. The considered chains are solved perfectly by the parser itself, and the LoRA fine-tuning does not directly relate to a realistic reasoning setting. Under replay, the parser already solves the chain itself; "extrapolation from 8 to 12 without error" is thus trivial.
  3. The stated "Loses its place" failure mode could be by construction; the nodes have no position index, and the static reader solves exactly that problem; it adds the idea of position back to the task.
  4. The headline 37.5 point-gap is measured with the full-context reader on path sets, which forms a train/test mismatch. The static reader was unstable over seeds, with constant LR, equal optimizer steps for the static reader could under train it, thus not overfitting. Tuning the reader would solve that.
  5. Nothing in the paper is diffusion-specific.
Questions for authors
  1. Could the authors emphasize what the principal scientific claim is? Is it a state-tracking mechanism or an evaluation methodology, and what experiment distinguishes it from prior work?
  2. Are the insights reproduced in other models and more realistic settings?
  3. An AR model could also receive one record at a time, or we rebuild its context, why do we solely consider a diffusion language model?
  4. What part of the reported gap is attributable to the training recipe?
  5. Could the authors give context to weakness (3)?
  6. Could the authors tune the static reader as well, to deal with the possible mismatch noted in (4)?
  7. Why is the paper focused on dLLM's, when the methodology does not appear to be specific to that class of models?
Reviewer_ukVwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper analyzes why revealing one evidence at a time to a masked diffusion language model (LLaDA-8B) helps on multi-hop pointer-chain tasks, separating the effects of what is visible when, what the paper call it as "reader" was trained on, and whether the read order is chosen during or before generation. A reader i.e., a finetuned LLaDA-8B is fine-tuned to see one evidence per block solves every depth-12 chain despite training only up to depth 8, while the best static reader given the same path records all at once reaches only 62.5%, mostly because it "loses its place" after a repeated digit and resumes from the wrong occurrence. The read order comes from a text parser that can compute the whole schedule before generation, so the gain reflects bookkeeping offloaded to the schedule rather than the model steering its own reads; in addition, the static baseline’s strength depends heavily on its training input, and on IMDB review chains with independent labels, classifying each review separately does at least as well as sequential reading.

Critical strengths

None

Critical weaknesses
  • Presentation is very poor and requires significant improvement.
    • Introduction section is very poorly written. It starts with "Figure 1a shows a prompt with sixteen records in shuffled order. Each record names a successor and carries a digit, and the answer is the sequence of digits along the chain that starts at node 8: 4 8 3 3". Please start your paragraph with a topic sentence.
    • Line 100 mentions "2.1M-parameter model" but there is no where in the manuscript that mentions this.
    • Line 353: MuSiQue-Ans dataset comes out from nowhere.
    • Line 366: "The honest summary [...]" is redundant (it can be simply stated as "The summary" to follow standard academic writing.) and looks very much AI-generated.
  • The experiments only used one diffusion LM i.e., LLaDA 8B. We therefore cannot tell does the proposed method generalizes towards other diffusion language models.
Questions for authors
  • What is the motivation of using only one diffusion language model i.e., LLaDA 8B and not other diffusion language models like SDAR [1]?

[1] https://arxiv.org/abs/2510.06303

Reviewer_SbJiOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how sequentially revealing evidence affects the performance of masked diffusion language models.

Critical strengths

None

Critical weaknesses
  1. I cannot find any motivation for this paper. The introduction begins with a toy example predefined by the authors. There is almost no motivation provided throughout the paper, and I can barely find any relevant references.

  2. The experimental results are extremely trivial. An external Python program handles the difficult pointer-tracking problem, and the only thing the model needs to do is print the numbers sequentially. I cannot find any interesting observations or insights from this experiment.

  3. I also cannot find any contribution of this paper. The paper itself does not specify its contributions in either the introduction or the conclusion.

Questions for authors

Although I am familiar with the field of discrete diffusion, I may not be familiar with the research area this paper addresses. Could the authors please clarify the motivation and contributions of this paper with references to relevant prior work, and explain why the experimental results should be considered nontrivial?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.