- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes GoldiMask, a supervised fine-tuning method for Masked Diffusion Language Models (MDLM), which jointly optimizes context selection and target weighting. The method first over-masks the response, then uses model-predicted probabilities and attention to select tokens to be revealed through submodular optimization. Subsequently, weights are assigned to the remaining training targets based on the changes in predicted probabilities before and after revealing and the token-level supervision priority. The authors evaluate GoldiMask on LLaDA and Dream series models using s1K, LIFT-SFT-12K, and KodCode, and compare it with vanilla SFT and various adaptive SFT methods. Experimental results show that GoldiMask achieves good average accuracy across multiple settings and reduces the number of decoding iterations in some parallel decoding settings. The paper also provides component ablation and computational cost analysis.
Critical strengths
-
GoldiMask jointly considers context selection and target weighting, proposing a reasonable optimization method to address the problem in diffusion SFT where the mask simultaneously determines both the available context and the supervised target. Compared to existing adaptive masking methods, this joint design is innovative.
-
Experiments cover three backbones and multiple training datasets, and baselines are retrained under uniform settings. Stable average gain across models, component ablation, and multi-seed results provide substantial support for the effectiveness of the method.
Critical weaknesses
- GoldiMask's core design lacks sufficient justification. Appendix H.5 shows that the predictive power of attention-based contextual support is weak (R^2 = 0.041 – 0.098), and it's unclear whether it truly captures token dependencies. Furthermore, in Table 4, random reveal + water-filling achieves 39.2, while the complete method achieves 41.1; other priority functions and weighting rules also perform similarly. Therefore, the necessity of complex submodular optimization and specific weighting design remains unclear.
- The evaluation protocol has ambiguities. Tables 1–2 do not specify whether the generation length is selected through the validation set, and the performance advantage is significantly reduced in some settings after fixing the length (Table 9). Additionally, some of the average accuracies of HumanEval and MBPP in Table 3 cannot be directly obtained from the standard test set sizes, a single greedy decoding, and three seeds; further explanation of the evaluation and statistical methods is needed.
Questions for authors
- How were the generation lengths chosen in Tables 1–2? If the length is chosen using a validation set, does the main performance advantage still hold?
- Could you elaborate on the evaluation protocols for HumanEval and MBPP, including the number of test items, decoding strategy, number of samples per item, and random seeds, to explain the average accuracy in Table 3?
- Appendix H shows that the prediction ability of the attention-based support proxy is relatively weak. Could you compare it with a simpler strategy that selects reveal tokens solely based on token probability, within the same two-pass pipeline?
- All component ablations are conducted on LLaDA-8B-Instruct with s1K, where random reveals with water-filling (39.2) already approach the strongest baseline LIFT (39.5). Could you report the same ablation on LLaDA-1.5 and Dream-7B to verify whether submodular reveal selection still clearly outperforms random reveals?
3 comments
Thanks for the detailed response and the additional experiments. The clarifications on generation length and the code evaluation, together with the new ablations, address my questions. I will keep my positive score.
We thank the reviewer for their comments, which have helped us improve our paper. We are open to answer any other questions in the reminder of the discussion period.
We thank the reviewer for the thoughtful evaluation of our method and experimental protocol. Below, we address the questions and describe the supporting analyses and revisions.
Q1. How were generation lengths chosen?
Following LIFT and d1, we evaluate each method at generation lengths of 128, 256, and 512 tokens and report the highest accuracy separately for each benchmark. Checkpoints are selected using validation loss; generation lengths are selected using benchmark accuracy, rather than the validation split. The same procedure is applied to all methods. We now clarify this in Appendix F in the updated manuscript.
Q2. HumanEval and MBPP evaluation protocol
We thank the reviewer for pointing this out. The three-seed protocol described for Tables 1-2 applied to the reasoning experiments, whereas the previous code-generation results in Table 3 averaged two training seeds. We have now added a third seed for every method and updated Table 3 and its caption to report means and standard deviations across three seeds.
We evaluate all 164 HumanEval problems and the 257 problems in the sanitized MBPP test split, using greedy diffusion decoding with generation length 256 and one sample per problem. A problem is counted as solved if the generated code passes all its unit tests.
The leading methods remain unchanged.
Q3+Q4. Additional ablation on LLaDA-1.5 and Dream-7B, with an additional probability-based selection method
We thank the reviewer for both suggestions. We address Q3 and Q4 together through additional component ablations on LLaDA-1.5 and Dream-7B, including the suggested probability-only selector. We ran the full component ablation on LLaDA-1.5 and Dream-7B (s1K, three seeds) and added the probability-only selector suggested by the reviewer. It uses the same two-pass pipeline and water-filling weights as GoldiMask-D, but selects reveal tokens by token probability alone, without the attention-based support. Entries are mean ± standard deviation over three seeds.
These results support the contribution of both context selection and target weighting. Random reveals with water-filling already improve over Vanilla SFT, while submodular selection provides further gains of 1.5 points on LLaDA-1.5 and 1.2 points on Dream-7B. GoldiMask-D also improves over probability-only selection by 0.7 and 0.9 points, respectively. We have included these ablations to Appendix I.3 of the revised manuscript. We also added Studies 4-5 in Appendix H.6-H.7. The selected reveal set outperforms all 16 random alternatives in 80.5% of tested states with t≤0.75, and mean prediction improvement increases across estimated-support bins. These analyses complement the downstream ablations by directly examining the benefit of revealed context.