acceptodds
Withdrawn from ICLR 2027

SlotWeave: Dependency-Aware Adaptive Slot Scheduling for Diffusion Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Masked diffusion large language models (dLLMs) enable parallel token generation, but independently decoding dependent positions can produce incoherent outputs. Slot-based diffusion decoding addresses this issue through autoregressive infilling within fixed-size slots and selecting decodable parallel slots by confidence. However, fixed slot boundaries can separate strongly coupled tokens, while confidence-based selection does not explicitly prevent mutually dependent slots from being decoded together. We propose SlotWeave, a training-free dependency-aware adaptive scheduler for slot-based dLLM decoding. SlotWeave estimates dependencies from attention weights obtained during the decoding forward pass. It first adaptively partitions remaining masked positions into contiguous slots by merging highly coupled neighbors. Then SlotWeave constructs a slot-level dependency graph and greedily selects a non-conflicting set of slots for parallel autoregressive infilling. Experiments on mathematical reasoning and code generation benchmarks show accuracy gains over the fixed slot baseline. For example, on MATH500 at primitive slot size , it improves accuracy from 49.4% to 53.2% with an approximately 10% reduction in throughput. Ablation studies further support the design choices of our method.

Primary area:
generative models
Keywords:
Diffusion language models, Dependency-aware decoding, Parallel generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uweQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SlotWeave, a training-free dependency-aware decoding method for slot-based diffusion language models like ReFusion. It estimates token dependencies from attention weights to adaptively merge strongly coupled slots and selects non-conflicting slots through a dependency graph for parallel autoregressive infilling. Experiments on different benchmarks show improved accuracy over ReFusion, especially with small primitive slot sizes, with moderate throughput degradation.

Critical strengths
  1. The key insight is relatively easy to understand and flow. Overall, the proposed method address a reasonable limitation of slot-based diffusion LLM, and the experiments showcase the method's effectiveness.
Critical weaknesses
  1. The method is built on top of ReFusion, which is the only one slot-based decoding dLLMs as far as I know. As a result, there is only one baseline direct baseline. There is no comparison, such as quality-speed tradeoff, against other dLLMs or autoregressive LLMs, which weakens my confidence in this work.

  2. In Appendix B.4's Table 4, the accuracy of left-to-right method is higher than SlotWeave in some cases. Then, Appendix B.6 notes that the τdep\tau_{\mathrm{dep}} begins from 0 and ends at 0.005, which means SlotWeave is close to left-to-right method, especially in the early stage of decoding. The contribution of dependency awareness here is not isolated.

Questions for authors
  1. Can you extend the evaluation with matched quality–speed comparisons against other dLLM decoding methods?

  2. Can you explain why there is a ρ\rho parameter in the dependency estimation? What happens if we just use all the layers instead of up to ρ=0.8\rho=0.8?

Reviewer_qxwaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an adaptive slot scheduling method based on attention-derived dependency estimates. By merging strongly dependent adjacent slots and using a dependency graph to constrain parallel decoding across slots, SlotWeave improves ReFusion's generation accuracy at smaller slot sizes, at the cost of some throughput.

Critical strengths

None

Critical weaknesses
  1. The main accuracy improvements reported in Table 1 are concentrated at smaller primitive slot sizes, particularly s=2s=2. However, the original ReFusion paper explicitly uses a default slot size of 32 for GSM8K, MATH, HumanEval, and MBPP (Appendix B.2, Table 5), while the slot sizes used during training are drawn from 4,8,16,32{4,8,16,32}, excluding s=2s=2. In SlotWeave's own Table 1, when s=32s=32, the improvements on GSM8K, MATH500, HumanEval, and MBPP are only 0.1, 0, 0, and 0 percentage points, respectively. Furthermore, Figure 4 of the original ReFusion paper shows that increasing the slot size can improve both accuracy and throughput on MBPP. Therefore, the current results indicate that the proposed adaptive slot scheduling offers no advantage over simply using ReFusion's original default configuration.
  2. SlotWeave's accuracy improvements come at the cost of reduced throughput, but the paper only reports TPS at s=8s=8 and does not provide a complete Accuracy–Throughput comparison across different slot sizes. In particular, ReFusion with its default s=32s=32 already achieves accuracy comparable to SlotWeave, so it remains unclear whether SlotWeave can achieve better accuracy at the same throughput. Furthermore, SlotWeave requires extracting attention weights from multiple Transformer layers to estimate token dependencies, but the paper does not specify the attention backend used for this process or whether it affects efficient attention kernels such as FlashAttention. Since the original ReFusion implementation supports efficient attention, while explicitly extracting attention weights may cause SDPA to fall back to eager attention, the kernel configurations and attention extraction overhead of both methods should be clarified to determine whether the reported throughput results reflect a fair comparison of practical inference efficiency.
Questions for authors

see Weaknesses

Reviewer_uTTLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a training-free decoding method that extends existing slot diffusion method ReFusion’s fixed-size, confidence-based slot scheduling. The method SlotWeave uses attention weights to estimate dependencies between masked token positions and merges strongly coupled neighboring slots, thus adatively creating larger slots. It then constructs a dependency graph between these slots and selects a non-conflicting subset for parallel decoding. Experiments on mathematical reasoning and code generation show accuracy improvements over ReFusion, particularly at small primitive slot sizes, with reduced throughput. However, the comparisons match initial rather than effective slot sizes, and the paper does not report the resulting adaptive slot-length distributions. Consequently, the results do not clearly establish how much of the improvement comes from dependency-aware scheduling versus increased autoregressive conditioning through larger slots and more conservative parallel decoding.

Critical strengths
  1. The approach is motivated appropriately, and the paper clearly explains how fixed slot boundaries and confidence-based selection can overlook token dependencies.

  2. The experiments cover several mathematical reasoning and code generation benchmarks, with consistent accuracy improvements over ReFusion at small primitive slot sizes.

Critical weaknesses
  1. The comparison in Table 1 does not adequately control for effective slot size. ReFusion retains the specified primitive slot size, whereas SlotWeave can merge these primitives into larger slots, pushing the decoding more to standard autoregression. The paper does not report the resulting slot-length distributions, making it difficult to determine whether the gains arise from dependency-aware partitioning or simply from increasing slot size. The diminishing gains and near-identical performance at larger primitive sizes point to the latter. Comparisons against fixed-size ReFusion with similar effective slot lengths would help establish the benefit of adaptive boundaries.

  2. The accuracy–efficiency advantage remains insufficiently established. Throughput is reported only at primitive slot size 8, whereas the largest accuracy gains occur at smaller sizes. Moreover, Appendix B.4 shows that simple left-to-right serialization outperforms SlotWeave on several benchmarks at primitive slot size 2, but provides no corresponding latency comparison. Accuracy–throughput curves covering larger fixed slots and more conservative ReFusion schedules are needed to establish whether SlotWeave offers a better trade-off.

Questions for authors

Please address the weaknesses above. Specifically I would like to know the following:

  1. Could you report the mean and distribution of merged-slot lengths for each primitive slot size and benchmark?
  2. How does SlotWeave compare with fixed-size ReFusion using slot lengths close to those averages?
  3. Could you report accuracy–throughput curves across all primitive slot sizes, including the left-to-right serialized baseline and more conservative ReFusion selection thresholds?
  4. Would simple merging strategies, such as random boundaries matched to SlotWeave’s slot-length distribution, achieve similar gains?

I would be happy to increase my score based on the authors repsonse.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.