- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes HiBlock, a hierarchical policy for learning input-dependent block sizes in diffusion LLMs. The method treats block size as a high-level action and jointly optimizes block-size selection and block-based response generation with a GRPO-style objective. During training, different block sizes are explored across rollouts, while at inference the learned policy selects one block size before generation. Experiments cover 13 benchmarks, 6 dLLM backbones, and several RL algorithms.
Critical strengths
-
The paper addresses a practical limitation of fixed-block dLLMs with a simple and useful idea: making block size an input-dependent learned action rather than a fixed hyperparameter.
-
The empirical study is fairly extensive. The improvements are consistent across multiple benchmarks, model backbones, RL algorithms, and the additional multi-seed experiment.
Critical weaknesses
-
The checkpoint-selection protocol is unclear and may affect the cross-domain claim. The paper states that the best checkpoint accuracy is reported, with checkpoints evaluated every 100 training steps, but I could not find how the best checkpoint is selected. If target test performance is used to select a checkpoint separately for each benchmark, this would introduce target-test model selection and weaken the claimed zero-shot cross-domain generalization.
-
The current ablations do not fully isolate the benefit of the learned block-size policy. HiBlock trains the response policy using rollouts from all candidate block sizes, while the main fixed-size baseline is trained only with block size 32. Thus, part of the gain may come from multi-block-size training itself rather than from the learned high-level selector. A stronger control would use the same multi-block-size training as HiBlock but remove the learned selector at inference. I would also like to see a fixed-block baseline whose block size is selected on source-domain validation rather than always using 32. This is particularly important because closely related work such as Block-R1 has already studied block-size conflict in multi-domain dLLM RL; the main novelty here appears to be the learned online selector rather than the conflict itself.
Questions for authors
-
How exactly is the "best checkpoint" selected for each reported target benchmark? Is any target test data used for checkpoint selection? Could you report results using a single checkpoint selected only from source-domain validation?
-
What happens if the low-level response policy is trained with exactly the same balanced multi-block-size exploration as HiBlock, but inference uses a fixed, random, or source-validation-selected block size instead of the learned high-level policy? This would help separate the effect of multi-block training from the effect of the learned selector.
-
During training, block-size actions are assigned by balanced exploration rather than sampled from the old high-level policy. What is the justification for using the current/old policy likelihood ratio for these action positions in Eq. 14? What objective should this update be interpreted as optimizing?
8 comments
Response to Reviewer exLG [2/2]
Re Q3: Current/Old Policy Likelihood Ratio for Block-Size Actions in Eq. 14
Thank you for raising this important theoretical question. We clarify our design motivation, experimental evidence, and optimisation objective as follows:
- We initially designed purely on-policy rollouts for both the high and low level policies.
- Our further analysis reveals that the initial block size policy exploration was insufficient, as shown below:
| Block Size | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|
| Without Balance | 52.4% | 8.5% | 18.6% | 5.2% | 12.5% | 2.8% |
The policy strongly favours block size 4 (52.4%), while block size 128 is rarely explored (2.8%). This limits the reward signals for other block sizes.
-
Thus, we introduce balanced exploration of block size actions to ensure sufficient exploration of all block sizes. This design is reasonable because:
- We can obtain the policy likelihood of each assigned block size action from the dLLM.
- The action itself does not need to be differentiable. Its likelihood provides the gradient for policy optimisation.
- Balanced sampling ensures that each block size is explored within a GRPO group.
-
We follow your suggestion to provide an additional ablation study to verify the effectiveness of balanced exploration:
| Training Strategy | Countdown | GSM8K | MATH500 | Sudoku | Avg. |
|---|---|---|---|---|---|
| With Balance (HiBlock) | 59.76 | 81.58 | 34.40 | 25.80 | 50.39 |
| Without Balance | 39.72 | 70.33 | 29.00 | 8.67 | 36.93 |
Our proposed balanced exploration improves significantly.
To further interpret this optimisation objective, we kindly provide that:
- We use balanced exploration to ensure sufficient coverage of candidate block sizes.
- The current/old policy likelihood ratio in Eq. 14 provides a proximal reference for advantage weighted policy updates.
- This ratio is not an importance sampling correction for the balanced exploration distribution.
- Thus, Eq. 14 defines a GRPO style clipped surrogate over balanced exploration trajectories, rather than the standard on-policy GRPO objective.
We sincerely thank you for this insightful question. We have clarified the objective in Section 3.3 Line 302 to 305 and have added this important ablation study in Section 4.3, Line 429 to 439. Your feedback has greatly helped us improve the manuscript.
We wholeheartedly thank you again for your constructive feedback, which has helped us significantly improve the quality of paper. If you have any further questions, we would be happy to discuss them with you.
I have one remaining question regarding Q3. While the balanced exploration ablation demonstrates its empirical benefit, it does not directly address the mismatch between the balanced block-size sampling distribution and the policy likelihood ratio in Eq. 14. Could the authors further clarify the theoretical justification for this surrogate objective? In particular, have you considered comparing it with an alternative objective that explicitly accounts for the behavior sampling distribution?
Response to Reviewer exLG [1/2]
We sincerely thank you for your constructive feedback. We appreciate your recognition of HiBlock's motivation and extensive empirical evaluation. We summarise our response and highlight the key information as follows:
Thank you for the question. We totally agree with your opinion.
Therefore, we would like to double confirm that our checkpoint selections:
We would like to provide more details:
Reporting the best checkpoint results based on validation set is a common practice in existing dLLM research. We quoted as below,
For all our experiments, 10% of training data is reserved for validation. This is used to evaluate checkpoints every 100 training steps.
We would like to reassure that (1) only validation data is used for model selection; (2) test data is absolutely prohibited in training or validation.
Thank you for this insightful suggestion. We totally agree with you on isolating the contribution of learnt block-size selection.
Following your suggestion, we conduct additional experiments with the same HiBlock checkpoint while changing the block size policy as follows:
We would like to respectfully clarify that the targeted problem and the technical contributions are fundamentally different between Block-R1 and our HiBlock.