- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies head importance in diffusion LLMs across denoising stages, using masking rate as a controlled proxy for stage. Empirically, rate-specific head rankings share structure at nearby rates but diverge at distant ones. The method, ARA, is an integrated-gradients-style head attribution rule that averages over paths with independently sampled per-head activation times rather than the synchronized diagonal path. It preserves completeness at each rate while giving full-dimensional support to the measure over gate configurations. A two-gate counterexample shows a case where the diagonal path scores a head at zero while ARA scores it 1/9. Experiments on LLaDA-1.5 and Dream-v0-Instruct-7B across eight benchmarks use prune-most and adaptive sparse-attention protocols against CoKV, CHG, and AttAttr, with ablations against DP (Dynamic) and ARA (Global).
Critical strengths
- The limitation of synchronized restoration is articulated correctly, and framing it as a path-design question rather than an estimator-budget one is useful. The two-gate counterexample makes the gap concrete.
- Theoretical grounding is appropriate. Completeness is preserved by construction. Proposition 5.3 formalizes full-dimensional support without overclaiming.
- The comparisons are fair. Wall-clock calibration for CoKV, matched gate-gradient evaluations between DP and ARA (one path of 32 steps vs four paths of 8), and convergence checks for CHG and AttAttr are correct. All methods use the same dynamic-sparse protocol.
- The DP (Dynamic) vs ARA (Dynamic) vs ARA (Global) decomposition cleanly separates path design from rate conditioning, and both contribute. ARA (Dynamic) beats DP (Dynamic) by 4.54 points on Dream and 3.90 on LLaDA on the macro-average, and beats ARA (Global) by 1.31 and 1.53, isolating the asynchronous path and rate conditioning respectively.
- Table 4 is a strength. Doubling K or P changes rankings only marginally, while independent four-path sets drop to 0.936, giving an honest picture of estimator noise.
Critical weaknesses
- Masking rate stands in for denoising stage, but real trajectories visit correlated model-selected masks, not necessarily uniform ones. No experiment compares uniform-mask attribution to attribution on realistic mid-generation masks at the matching rate.
- Four paths may be undersized for H in the thousands. LLaDA has 1,024 query heads, Dream has 784, yet P = 4. Independent four-path sets agree at only about 0.94 Spearman, and ARA (Dynamic) in Table 2 shows per-task standard deviations as large as 3.00 (GSM8K on LLaDA) and 2.50 (GSM8K on Dream), which blur several of the per-task gaps.
- Table 2 standard deviations more broadly undermine per-task significance across variants. DP (Dynamic) on CMMLU for LLaDA has sigma 8.40, and several other cells sit above sigma 2. Only two runs per cell. Three or more seeds and a per-task significance test would make the Dynamic vs Global comparison far more defensible.
- The scale is limited to two models in two families with no scaling study or cross-size check within a family.
- The paper flags that opposite-signed per-path contributions can cancel but doesn’t report per-path variance for any heads.
- No comparison against a plain expected-gradients baseline that samples gate configurations uniformly in [0,1]^H. That would isolate the specific choice of ramp schedules.
Questions for authors
- Does ARA on uniform masks agree with ARA on masks drawn from real mid-generation trajectories at the matching rate?
- How does ARA compare to expected gradients with uniformly sampled gate configurations?
- What is the per-path variance of contributions for a representative set of heads at P = 4?
- Add error bars to Figure 3. Is the 20 to 50 percent lead within noise?
- On LLaDA’s shift toward shallower top layers at low masking rates, does the shift reverse or saturate below 0.15?
- What is the wall-clock overhead of rate-specific precomputed attribution in a production sparse-attention setup?
2 comments
Response to W5 and Q3: Per-path variance. Thank you for pointing this out. We analyzed the individual path contributions for LLaDA on GSM8K and Minerva Math. For each example, masking rate, and head, we computed the variance across the four paths, then averaged these variances over all examples, rates, and heads. We also counted whether the four contributions had the same sign, a 3:1 sign split, or a 2:2 sign split. The four paths agree in sign in over 80% of cases on both tasks. Opposite-signed contributions occur in 19.29% of GSM8K cases and 15.49% of Minerva Math cases. ARA averages signed contributions, and its score reflects their net effect across restoration paths.
| Task | Mean path variance | Same sign | 3:1 split | 2:2 split |
|---|---|---|---|---|
| GSM8K | 0.001357 | 80.71% | 13.43% | 5.86% |
| Minerva Math | 0.001251 | 84.51% | 11.36% | 4.13% |
Response to W6 and Q2: Uniform gate sampling. Thank you for this suggestion. Uniformly sampling gate configurations and averaging their gradients gives equal weight to g and 1-g. For an interaction term satisfying F(1-g)=F(g), the gradients at these configurations have opposite signs, so its contribution cancels in expectation. Such an interaction can still affect the model near the fully restored state. ARA's delayed-start restoration schedules do not have this reflection symmetry, so this cancellation is not enforced. In the smooth, monotone example in Appendix A.8, uniform gradient averaging ranks the heads as 3 > 2 > 1. ARA instead gives 1 > 3 > 2 in the exact path expectation, matching the effects of both small gate reductions and complete suppression. This gives a concrete reason for the restoration schedule: it can retain interaction contributions that uniform averaging cancels. We compare ARA with DP under matched computational budgets in Table 2. We also added an SPI-style random-path baseline to Table 1 for this revision. SPI uses stick-breaking schedules rather than ARA's delayed-start ramps, providing a comparison with a different random-path design under the same attribution budget.
Response to Q4: Error bars for Figure 3. Thank you for raising this concern. We added a second seed at 30% retained blocks to check the stability of Figure 3, comparing ARA with the strongest baseline for each model. A full run covers five retained-block ratios and requires 2 models × 4 methods × 5 ratios × 8 datasets = 320 dataset evaluations, five times the workload at one ratio. Given the rebuttal time and compute limits, we focused the additional run on 30% retention. Each point in Figure 3 averages eight tasks. Tables 1 and 2 show that these macro averages can vary less across seeds than individual task scores. At 30% retention, the eight-task averages over two seeds are below (mean ± population SD). ARA leads by 1.61 points on Dream and 0.71 on LLaDA, exceeding the SD of either method in each comparison. It remains ahead in both runs on both models. These results support the stability of ARA's advantage at 30% retention and show that the gain is not specific to the original run.
| Model | ARA | Strongest baseline |
|---|---|---|
| Dream | 41.87 ± 0.25 | CoKV: 40.26 ± 0.18 |
| LLaDA | 48.46 ± 0.30 | CHG: 47.75 ± 0.02 |
Response to Q5. Behavior below a masking rate of 0.15. We extended the LLaDA analysis to masking rates of 0.10 and 0.05. The mean layer indices of the 103 highest-ranked heads are shown below, alongside the original result at 0.15. Lower indices indicate shallower layers. The changes below 0.15 are small and non-monotonic in both tasks. We therefore cannot draw a firm conclusion that the shift reverses or saturates. The added rates are only 0.05 apart and cover a much narrower interval than our original grid of 0.90, 0.70, 0.50, 0.30, and 0.15. Our observation of a shift toward shallower layers is supported over that broader range. These additional results do not establish a clear trend below 0.15.
| Masking rate | Code | Math |
|---|---|---|
| 0.15 | 21.08 | 23.05 |
| 0.10 | 21.13 | 22.92 |
| 0.05 | 21.06 | 23.11 |
Response to Q6. Attribution overhead. Our recorded attribution runs used H100 hardware, 200 examples per task, five masking rates, two masks per rate, four paths, and eight integration steps per path. The table below reports wall-clock times in minutes. This is a one-time preprocessing cost of about 1.1-5.5 hours per model and task. For the same model and calibration setup, the stored scores can guide adaptive sparse attention for all subsequent inference requests without recomputing attribution. No attribution gradients are computed during inference. We have added these settings, timings, and the reuse explanation to Appendix A.1 (Table 5) of the revised paper.
| Model | MMLU | GSM8K | Minerva Math |
|---|---|---|---|
| Dream | 131.43 | 135.32 | 330.55 |
| LLaDA | 67.40 | 154.22 | 184.28 |
Thank you for your insightful and helpful suggestions, and we hope our point-by-point responses below address your concerns.
Response to W1 and Q1. Random masks and real decoding states. Thank you for raising this question. We compared ARA with random masks against ARA on cached intermediate decoding states. For the latter, we preserved the generated tokens and model-selected mask positions from decoding. In our main setup, random masks are applied to complete model-generated responses. Both setups use generated tokens as targets. Results below use LLaDA and Table 1's pruning and evaluation settings (mean ± population SD over two runs). Lower scores after head removal indicate more influential selected heads.
ARA with random masks gives lower mean scores on all four tasks. A single decoding trajectory usually provides only one state at a given rate. Random masking lets us reuse each generated response to create several attribution inputs at the same masking rate. Each mask leaves a different subset of tokens visible, allowing us to assess head contributions under different contexts. We average the attributions over these different states. We chose this design to reduce the need to collect additional generated responses and make attribution more practical with limited calibration data.
Response to W2 and W3. Stability across seeds. Thank you for raising this concern. In Table 4 (Appendix A.1), independent four-path estimates on LLaDA have a mean Spearman correlation of 0.936, indicating stable head rankings. Doubling P from 4 to 8 changes the rankings very little, with a correlation of 0.983. This suggests limited additional benefit from more paths in this setting. Small changes near the selection cutoff can still change the removed heads and affect generated tokens and later decoding states. This can cause larger score differences on individual tasks even when overall rankings are similar. Following your suggestion, we added a third seed for every variant on both models and updated Table 2 and the analysis in Section 6.5 of the revised paper. Below are eight-task macro averages (mean ± population SD across three seeds). Lower is better.
ARA (Dynamic) gives the lowest macro-average in every run for both models. Compared with DP (Dynamic), it lowers the mean by 4.42 points on Dream and 3.96 on LLaDA. Compared with ARA (Global), the reductions are 1.18 and 1.99 points. Its macro-average SDs are 0.67 and 0.73, smaller than these mean gaps. These results support a consistent overall benefit from asynchronous paths and rate-specific head selection despite variation on individual tasks.
Response to W4: Scaling to a larger model. We added a prune-most-important-head experiment on LLaDA2.0-16B. Given the rebuttal time limit, we focused on GSM8K, where our earlier experiments showed clear differences among attribution methods. The dense model scores 86.5%. We remove 10%-30% of the top-ranked heads per layer, with all other attribution and evaluation settings unchanged from LLaDA-1.5. Lower accuracy after pruning indicates that the selected heads have a greater effect on task performance. ARA gives the lowest accuracy at three of the five removal ratios and the second-lowest at the other two. At the 20% ratio used for LLaDA-1.5, ARA lowers accuracy to 44.5%, compared with 48.5% for DP (Dynamic). This shows that its asynchronous paths remain useful for identifying influential heads in the larger model. ARA also computes gradients for all head gates jointly in each backward pass. With the calibration size, masking rates, paths, and integration steps fixed, more heads do not require additional gradient evaluations. The cost of each evaluation follows the larger model's forward and backward computation. This design, together with the 16B results, supports the scalability of ARA to larger DLLMs.