- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The author provides a new framework regarding the controllability of the DLMs. It uses an external anchor generator to generate a {"Position": "Content"} to guide the decoding of the DLMs. The method has good performance on the selected metrics.
Critical strengths
-
The paper uses something AR models can’t do. It exploits DLMs’ bidirectional denoising to place guidance anywhere in the output, not just at the start.
-
The approach is model-agnostic.
-
The controllability problem is good and important.
Critical weaknesses
-
The most important concern for me is the experimental part. The vanilla/prompt accuracies are well below the published numbers (especially LLaDA; refer to the LLaDA paper), and LLaDA-Ins scores are below LLaDA-Base on GSM8K. The numbers can vary in the actual experiments, which is understandable, but shouldn't be so different. No decoding configuration is reported: generation length, steps, few-shot examples, chat template. I assume no block diffusion is applied. What decoding strategies does LLaDA/Dream use? Low-confidence and high-entropy?
-
Comparison is weak, and the reasoning gain is minimal. Given there is no variability (+- std) reported, a lot of the performance gain can be simply attributed to randomness. How many seeds are reported for the main experiments? I assume better controllability can somehow lift the accuracy performance. If not, what is the point anyway? AnchorDLM gets Claude Opus 4.6 rewritten training data, SFT, and RL with 8 DLM rollouts per query. All baselines (Prompt, S3, Ti) are training-free. A trained baseline is missing, e.g., LoRA fine-tuning the DLM on the same Claude-rewritten data, or few-shot prompting with examples in the target style.
-
Anchor could just leak answers. Table 2 itself shows that anchors from bigger models increase pass@1. So gains may come from Qwen’s or Claude’s reasoning being written into the answer, not from better control of the DLM.
Questions for authors
-
How much of AnchorDLM’s gain comes from the learned position rather than the anchor content? Please report an ablation that keeps the generated content but places it (a) at a fixed position, e.g., p = 0 as in SFT Stage 1, (b) at a random position, or (c) at the position the DLM would naturally produce that content. Also, how is the predicted position index mapped to the DLM’s token positions, given that Qwen and LLaDA/Dream use different tokenizers? According to Figure 3, the anchor position does not have a significant effect on G-mean?
-
An analysis of the anchor's actual outputs?
-
Prompt setting: What is the prompt? ICL with answer formatting is a natural baseline to compare with, so it should be included.
-
The most important flaw is the experiment's authenticity, and this directly affects my evaluation of this paper. If the author can resolve this, I am willing to raise my evaluation.
3 comments
Controllability and accuracy. Control comes at a small accuracy cost, which we report separately below.
| Backbone | Method | HumanEval-cmt Pass@1 | HumanEval-cmt Control Score | HumanEval-cmt G-mean | MBPP-cmt Pass@1 | MBPP-cmt Control Score | MBPP-cmt G-mean |
|---|---|---|---|---|---|---|---|
| LLaDA-8B-Base | Vanilla | 34.15 | 6.23 | 14.59 | 39.80 | 22.27 | 29.77 |
| Prompt | 31.71 | 17.10 | 23.29 | 37.40 | 25.96 | 31.16 | |
| S³ | 14.63 | 21.41 | 17.70 | 20.60 | 44.56 | 30.30 | |
| Ti | 15.24 | 18.65 | 16.86 | 21.20 | 42.77 | 30.11 | |
| AnchorDLM | 30.49 | 63.17 | 43.89 | 35.00 | 45.83 | 40.05 | |
| LLaDA-8B-Instruct | Vanilla | 39.02 | 22.80 | 29.83 | 40.80 | 19.56 | 28.25 |
| Prompt | 34.76 | 59.92 | 45.64 | 40.80 | 57.29 | 48.35 | |
| S³ | 37.80 | 39.76 | 38.77 | 31.80 | 34.47 | 33.11 | |
| Ti | 34.76 | 40.75 | 37.64 | 24.00 | 36.31 | 29.52 | |
| AnchorDLM | 34.15 | 62.16 | 46.07 | 36.40 | 68.56 | 49.96 |
Overall, AnchorDLM achieves a good balance between accuracy and controllability, with the highest G-mean in all four settings.
Reply to Question 3:
We used the backbone’s released in-context learning template and rewrote the demonstration answers in the target style. The few-shot example below is one of the four demonstrations we used to evaluate LLaDA-8B-Instruct on GSM8K-bwd. Before performing any arithmetic, the answer works backward from the final total to the original item counts, tracing the number of cards to the number of frisbees and then to the known number of marbles.
dict(
role='HUMAN',
prompt="Question: Bella has two times as many marbles as frisbees. She also has 20 more frisbees than deck cards. If she buys 2/5 times more of each item, what would be the total number of the items she will have if she currently has 60 marbles?\nLet's think step by step\nAnswer:",
),
dict(
role='BOT',
prompt="To find Bella's total after the purchase, we need her final count of each item. To find each final count, we need its original count so we can add 2/5 more. To find the original deck card count, we need the frisbee count. To find the frisbee count, we can work from the 60 marbles given in the question.\nBella has twice as many marbles as frisbees, so she has 60/2 = 30 frisbees. She has 20 more frisbees than deck cards, so she has 30-20 = 10 deck cards.\nShe buys 2/5*60 = 24 more marbles, giving 60+24 = 84 marbles. She buys 2/5*30 = 12 more frisbees, giving 30+12 = 42 frisbees. She buys 2/5*10 = 4 more deck cards, giving 10+4 = 14 deck cards.\nTogether, Bella will have a total of 84+42+14 = 140 items.\nThe answer is 140\n",
),
Reply to Question 4 and Weakness 1:
Thanks for pointing this out.
All results use full-sequence (pure) diffusion sampling; no block diffusion is applied. We use a generation length of 256 with 128 denoising steps, i.e. two tokens per step as in d1 [2], and each backbone's official unmasking strategy: low-confidence remasking for LLaDA and entropy-based unmasking for Dream.
To reduce interference from the heuristic that lowers EOS token confidence [1], we used full-sequence diffusion decoding with this heuristic disabled, so that changes in decoding order are attributable to the anchor alone. The lower GSM8K accuracy of LLaDA-8B-Instruct under this setting is consistent with the explanation in Appendix B.4 of LLaDA [1]. Heavy EOS padding during supervised fine-tuning can cause premature termination under pure diffusion sampling, shortening responses and reducing accuracy.
We now report both settings:
| Model / Method (EOS confidence suppression) | GSM8K-fmt G | HumanEval-cmt G | MBPP-str G | GSM8K Acc. | HumanEval pass@1 | MBPP pass@1 |
|---|---|---|---|---|---|---|
| LLaDA-8B-Base · Vanilla | 5.01 | 14.59 | 43.16 | 66.03 | 34.15 | 39.80 |
| LLaDA-8B-Instruct · Vanilla (off) | 17.08 | 29.83 | 32.06 | 56.10 | 39.02 | 40.80 |
| LLaDA-8B-Instruct · AnchorDLM (off) | 61.85 | 46.07 | 50.90 | 58.53 | 41.46 | 41.40 |
| LLaDA-8B-Instruct · Vanilla (on) | 19.30 | 31.37 | 32.27 | 66.19 | 42.07 | 41.00 |
| LLaDA-8B-Instruct · AnchorDLM (on) | 66.41 | 48.15 | 52.86 | 67.17 | 44.51 | 43.20 |
As shown in the table, enabling EOS confidence suppression improves the vanilla baseline. With EOS confidence suppression enabled, AnchorDLM outperforms baselines, demonstrating that our method remains effective under this decoding setting.
Reply to Weakness 3:
There is no test-answer leakage in our anchor training or evaluation procedure. Claude was used exclusively to rewrite and annotate training examples. All supervised anchors and preference pairs were constructed from training data. At test time, the 1.7B generator receives only the question and predicts an anchor without access to Claude or the reference solution. For mathematical reasoning tasks that require a final numerical answer, our annotation instructions explicitly exclude the final answer itself when selecting the anchor (Appendix A.2). To limit the amount of guidance provided to the DLM, we use short anchor spans. Across benchmarks, these spans account for 4.33%–10.88% of the generated tokens on average (Table 4).
Table 2 examines how generator capacity affects downstream performance. Increasing the generator size from 0.6B to 1.7B improves pass@1 on both benchmarks, while larger generators yield inconsistent further gains. We therefore selected 1.7B to balance anchor quality and inference cost.
References
[1] Nie et al. Large Language Diffusion Models. NeurIPS, 2025. Appendix B.4 and official OpenCompass evaluation configuration.
[2] Zhao et al. d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning. NeurIPS, 2025. Appendix D.
Thank you for raising these questions about anchor placement, baseline strength, and evaluation. Moving the same anchor to a fixed or random position lowers all six reported scores. EOS handling explains a substantial part of the GSM8K baseline gap, and AnchorDLM outperforms vanilla LLaDA-Instruct under both EOS settings. We also clarify how we prevent test-answer leakage: Claude annotates only training examples, and the trained anchor generator receives only the question at test time.
We hope these clarifications and additional results address your concerns, and we would be grateful if you would consider raising your score. We are happy to answer any further questions.
Reply to Questions 1–2 and Weakness 2:
Anchor placement. In Figure 3, we held the generated content C fixed and added an offset to its predicted position P at inference time. To test whether the predicted position matters more broadly, we used the trained generator to produce (P,C) once per query and placed the same content at the predicted position, at P=0, or at a random position. Here, P is a zero-based index into the target DLM’s masked output sequence, and P=0 denotes the first output token. The generator emits P as an integer and C as text. The target DLM’s tokenizer determines the token span occupied by C.
Both interventions lower all six scores, although sensitivity varies by task. Moving the same anchor to the prefix lowers HumanEval-cmt G-mean by 5.21 points and HumanEval pass@1 by 32.32 points. Because the anchor content is identical across placements for each query, these differences isolate the contribution of placement and support adapting it to the query and the generated span.
Actual anchor outputs. Figure 5 illustrates what an anchor contributes on MBPP task 389, which asks the model to count distinct unordered pairs. The reasoning anchor supplies the condition
if (first, second) in result:. The DLM completes the surrounding logic, including ordering each pair before the lookup and updating the dictionary. The unanchored completion counts reversed pairs separately. This example shows both the algorithmic information supplied by the anchor and the work performed by the backbone.Variability and trained baseline. The original experiments used a single seed. We evaluated all methods on LLaDA-8B-Base across three seeds and report means ± sample standard deviations. We also added a baseline that directly fine-tunes LLaDA-8B-Base with LoRA to generate the full Claude-rewritten solutions for the same training questions used to construct the supervised anchor targets. G-mean evaluates controlled generation, while GSM8K accuracy and MBPP pass@1 evaluate the reasoning configurations.
AnchorDLM achieves the highest mean on all four metrics, including against the new LoRA baseline. These results support our conclusion on the two benchmarks tested: AnchorDLM improves controlled generation, with the largest gain on GSM8K format control, and yields more modest gains in reasoning performance.