acceptodds
Under review as a conference paper at ICLR 2027

ARA: Asynchronous Restoration Attribution for Attention Heads in Diffusion Language Models

Abstract

Diffusion large language models (DLLMs) reconstruct masked tokens through successive denoising stages, yet the importance of their attention heads throughout this process remains unclear. We study how head importance changes with the available context, using masking rate as a controlled proxy for denoising stage. Head rankings retain shared structure but generally become less similar as masking rates move farther apart, revealing structured variation that a single global ranking can obscure. To quantify head contributions under these different reconstruction conditions, we introduce Asynchronous Restoration Attribution (ARA). The method averages contributions over paths that gradually restore suppressed head outputs according to independently sampled schedules. Unlike synchronized restoration, it evaluates each head while others are restored to different degrees. This allows it to capture interaction-dependent contributions that a synchronized path can miss. Experiments on two DLLM families across eight tasks support the effectiveness of ARA in identifying functionally important heads through dynamic pruning and guiding adaptive sparse attention.

Primary area:
generative models
Keywords:
diffusion large language model, attention head attribution

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_tjkJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies head importance in diffusion LLMs across denoising stages, using masking rate as a controlled proxy for stage. Empirically, rate-specific head rankings share structure at nearby rates but diverge at distant ones. The method, ARA, is an integrated-gradients-style head attribution rule that averages over paths with independently sampled per-head activation times rather than the synchronized diagonal path. It preserves completeness at each rate while giving full-dimensional support to the measure over gate configurations. A two-gate counterexample shows a case where the diagonal path scores a head at zero while ARA scores it 1/9. Experiments on LLaDA-1.5 and Dream-v0-Instruct-7B across eight benchmarks use prune-most and adaptive sparse-attention protocols against CoKV, CHG, and AttAttr, with ablations against DP (Dynamic) and ARA (Global).

Critical strengths
  • The limitation of synchronized restoration is articulated correctly, and framing it as a path-design question rather than an estimator-budget one is useful. The two-gate counterexample makes the gap concrete.
  • Theoretical grounding is appropriate. Completeness is preserved by construction. Proposition 5.3 formalizes full-dimensional support without overclaiming.
  • The comparisons are fair. Wall-clock calibration for CoKV, matched gate-gradient evaluations between DP and ARA (one path of 32 steps vs four paths of 8), and convergence checks for CHG and AttAttr are correct. All methods use the same dynamic-sparse protocol.
  • The DP (Dynamic) vs ARA (Dynamic) vs ARA (Global) decomposition cleanly separates path design from rate conditioning, and both contribute. ARA (Dynamic) beats DP (Dynamic) by 4.54 points on Dream and 3.90 on LLaDA on the macro-average, and beats ARA (Global) by 1.31 and 1.53, isolating the asynchronous path and rate conditioning respectively.
  • Table 4 is a strength. Doubling K or P changes rankings only marginally, while independent four-path sets drop to 0.936, giving an honest picture of estimator noise.
Critical weaknesses
  • Masking rate stands in for denoising stage, but real trajectories visit correlated model-selected masks, not necessarily uniform ones. No experiment compares uniform-mask attribution to attribution on realistic mid-generation masks at the matching rate.
  • Four paths may be undersized for H in the thousands. LLaDA has 1,024 query heads, Dream has 784, yet P = 4. Independent four-path sets agree at only about 0.94 Spearman, and ARA (Dynamic) in Table 2 shows per-task standard deviations as large as 3.00 (GSM8K on LLaDA) and 2.50 (GSM8K on Dream), which blur several of the per-task gaps.
  • Table 2 standard deviations more broadly undermine per-task significance across variants. DP (Dynamic) on CMMLU for LLaDA has sigma 8.40, and several other cells sit above sigma 2. Only two runs per cell. Three or more seeds and a per-task significance test would make the Dynamic vs Global comparison far more defensible.
  • The scale is limited to two models in two families with no scaling study or cross-size check within a family.
  • The paper flags that opposite-signed per-path contributions can cancel but doesn’t report per-path variance for any heads.
  • No comparison against a plain expected-gradients baseline that samples gate configurations uniformly in [0,1]^H. That would isolate the specific choice of ramp schedules.
Questions for authors
  • Does ARA on uniform masks agree with ARA on masks drawn from real mid-generation trajectories at the matching rate?
  • How does ARA compare to expected gradients with uniformly sampled gate configurations?
  • What is the per-path variance of contributions for a representative set of heads at P = 4?
  • Add error bars to Figure 3. Is the 20 to 50 percent lead within noise?
  • On LLaDA’s shift toward shallower top layers at low masking rates, does the shift reverse or saturate below 0.15?
  • What is the wall-clock overhead of rate-specific precomputed attribution in a production sparse-attention setup?
2 comments

Thank you for your insightful and helpful suggestions, and we hope our point-by-point responses below address your concerns.

Response to W1 and Q1. Random masks and real decoding states. Thank you for raising this question. We compared ARA with random masks against ARA on cached intermediate decoding states. For the latter, we preserved the generated tokens and model-selected mask positions from decoding. In our main setup, random masks are applied to complete model-generated responses. Both setups use generated tokens as targets. Results below use LLaDA and Table 1's pruning and evaluation settings (mean ± population SD over two runs). Lower scores after head removal indicate more influential selected heads.

SetupMMLUCMMLUC-EvalGPQA
ARA (real decoding states)47.24 ± 0.9241.83 ± 1.1641.60 ± 2.0125.50 ± 1.00
ARA (random masks, Table 1)44.54 ± 1.9135.09 ± 0.1732.91 ± 1.9325.25 ± 0.25

ARA with random masks gives lower mean scores on all four tasks. A single decoding trajectory usually provides only one state at a given rate. Random masking lets us reuse each generated response to create several attribution inputs at the same masking rate. Each mask leaves a different subset of tokens visible, allowing us to assess head contributions under different contexts. We average the attributions over these different states. We chose this design to reduce the need to collect additional generated responses and make attribution more practical with limited calibration data.

Response to W2 and W3. Stability across seeds. Thank you for raising this concern. In Table 4 (Appendix A.1), independent four-path estimates on LLaDA have a mean Spearman correlation of 0.936, indicating stable head rankings. Doubling P from 4 to 8 changes the rankings very little, with a correlation of 0.983. This suggests limited additional benefit from more paths in this setting. Small changes near the selection cutoff can still change the removed heads and affect generated tokens and later decoding states. This can cause larger score differences on individual tasks even when overall rankings are similar. Following your suggestion, we added a third seed for every variant on both models and updated Table 2 and the analysis in Section 6.5 of the revised paper. Below are eight-task macro averages (mean ± population SD across three seeds). Lower is better.

VariantDreamLLaDA
DP (Dynamic)35.08 ± 0.0535.28 ± 0.90
ARA (Global)31.85 ± 0.2833.31 ± 0.88
ARA (Dynamic)30.67 ± 0.6731.32 ± 0.73

ARA (Dynamic) gives the lowest macro-average in every run for both models. Compared with DP (Dynamic), it lowers the mean by 4.42 points on Dream and 3.96 on LLaDA. Compared with ARA (Global), the reductions are 1.18 and 1.99 points. Its macro-average SDs are 0.67 and 0.73, smaller than these mean gaps. These results support a consistent overall benefit from asynchronous paths and rate-specific head selection despite variation on individual tasks.

Response to W4: Scaling to a larger model. We added a prune-most-important-head experiment on LLaDA2.0-16B. Given the rebuttal time limit, we focused on GSM8K, where our earlier experiments showed clear differences among attribution methods. The dense model scores 86.5%. We remove 10%-30% of the top-ranked heads per layer, with all other attribution and evaluation settings unchanged from LLaDA-1.5. Lower accuracy after pruning indicates that the selected heads have a greater effect on task performance. ARA gives the lowest accuracy at three of the five removal ratios and the second-lowest at the other two. At the 20% ratio used for LLaDA-1.5, ARA lowers accuracy to 44.5%, compared with 48.5% for DP (Dynamic). This shows that its asynchronous paths remain useful for identifying influential heads in the larger model. ARA also computes gradients for all head gates jointly in each backward pass. With the calibration size, masking rates, paths, and integration steps fixed, more heads do not require additional gradient evaluations. The cost of each evaluation follows the larger model's forward and backward computation. This design, together with the 16B results, supports the scalability of ARA to larger DLLMs.

Method10%15%20%25%30%
AttAttr59.560.052.047.040.5
CoKV57.052.550.544.038.5
CHG52.551.045.545.530.0
DP (Dynamic)54.047.548.541.033.0
ARA (Dynamic)51.549.544.538.031.5

Response to W5 and Q3: Per-path variance. Thank you for pointing this out. We analyzed the individual path contributions for LLaDA on GSM8K and Minerva Math. For each example, masking rate, and head, we computed the variance across the four paths, then averaged these variances over all examples, rates, and heads. We also counted whether the four contributions had the same sign, a 3:1 sign split, or a 2:2 sign split. The four paths agree in sign in over 80% of cases on both tasks. Opposite-signed contributions occur in 19.29% of GSM8K cases and 15.49% of Minerva Math cases. ARA averages signed contributions, and its score reflects their net effect across restoration paths.

TaskMean path varianceSame sign3:1 split2:2 split
GSM8K0.00135780.71%13.43%5.86%
Minerva Math0.00125184.51%11.36%4.13%

Response to W6 and Q2: Uniform gate sampling. Thank you for this suggestion. Uniformly sampling gate configurations and averaging their gradients gives equal weight to g and 1-g. For an interaction term satisfying F(1-g)=F(g), the gradients at these configurations have opposite signs, so its contribution cancels in expectation. Such an interaction can still affect the model near the fully restored state. ARA's delayed-start restoration schedules do not have this reflection symmetry, so this cancellation is not enforced. In the smooth, monotone example in Appendix A.8, uniform gradient averaging ranks the heads as 3 > 2 > 1. ARA instead gives 1 > 3 > 2 in the exact path expectation, matching the effects of both small gate reductions and complete suppression. This gives a concrete reason for the restoration schedule: it can retain interaction contributions that uniform averaging cancels. We compare ARA with DP under matched computational budgets in Table 2. We also added an SPI-style random-path baseline to Table 1 for this revision. SPI uses stick-breaking schedules rather than ARA's delayed-start ramps, providing a comparison with a different random-path design under the same attribution budget.

Response to Q4: Error bars for Figure 3. Thank you for raising this concern. We added a second seed at 30% retained blocks to check the stability of Figure 3, comparing ARA with the strongest baseline for each model. A full run covers five retained-block ratios and requires 2 models × 4 methods × 5 ratios × 8 datasets = 320 dataset evaluations, five times the workload at one ratio. Given the rebuttal time and compute limits, we focused the additional run on 30% retention. Each point in Figure 3 averages eight tasks. Tables 1 and 2 show that these macro averages can vary less across seeds than individual task scores. At 30% retention, the eight-task averages over two seeds are below (mean ± population SD). ARA leads by 1.61 points on Dream and 0.71 on LLaDA, exceeding the SD of either method in each comparison. It remains ahead in both runs on both models. These results support the stability of ARA's advantage at 30% retention and show that the gain is not specific to the original run.

ModelARAStrongest baseline
Dream41.87 ± 0.25CoKV: 40.26 ± 0.18
LLaDA48.46 ± 0.30CHG: 47.75 ± 0.02

Response to Q5. Behavior below a masking rate of 0.15. We extended the LLaDA analysis to masking rates of 0.10 and 0.05. The mean layer indices of the 103 highest-ranked heads are shown below, alongside the original result at 0.15. Lower indices indicate shallower layers. The changes below 0.15 are small and non-monotonic in both tasks. We therefore cannot draw a firm conclusion that the shift reverses or saturates. The added rates are only 0.05 apart and cover a much narrower interval than our original grid of 0.90, 0.70, 0.50, 0.30, and 0.15. Our observation of a shift toward shallower layers is supported over that broader range. These additional results do not establish a clear trend below 0.15.

Masking rateCodeMath
0.1521.0823.05
0.1021.1322.92
0.0521.0623.11

Response to Q6. Attribution overhead. Our recorded attribution runs used H100 hardware, 200 examples per task, five masking rates, two masks per rate, four paths, and eight integration steps per path. The table below reports wall-clock times in minutes. This is a one-time preprocessing cost of about 1.1-5.5 hours per model and task. For the same model and calibration setup, the stored scores can guide adaptive sparse attention for all subsequent inference requests without recomputing attribution. No attribution gradients are computed during inference. We have added these settings, timings, and the reuse explanation to Appendix A.1 (Table 5) of the revised paper.

ModelMMLUGSM8KMinerva Math
Dream131.43135.32330.55
LLaDA67.40154.22184.28
Reviewer_8XEnOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies attention head contributions over the course of denoising process in Diffusion Language Models. The paper proposes a new path-based attribution method for measuring the contribution of each head. At each masking rate, the method modulates the restoration strength of each head independently of the others. This way, the method allows to study complementary or substitutable effects of different heads. Using the proposed method, the paper then studies how the contributions of different heads vary over the unmasking process in diffusion language models. They validate the effectiveness of their proposed method through sparse attention experiments, and pruning most important heads.

Critical strengths
  • The paper is relatively easy to follow, with helpful examples allowing to see the discussed problems in practice. The proposed method is well-motivated and explained.
  • The proposed method is well-validated, across two models and 8 benchmarks, and experiments relying both on removing the attention heads considered as most important (which should lead to significant performance drop) and maintaining only the most important heads (where the performance should stay as close to the baseline performance as possible). The additional ablation showing the usefulness of the dynamic approach is also insightful.
Critical weaknesses
  1. While the primary focus and motivation of the paper seems to be studying diffusion language models, the proposed method (ARA) does not seem to be particularly tailored to DLMs, but rather looks like it could be applied to any attention-based model. If this is the case, I would encourage the authors to also analyse its effectiveness in the autoregressive setting. If this is not the case, then could the authors please explain what makes the method particularly tailored to the DLM setting?
  2. I think if the authors want to focus the paper on DLMs, it would also be nice to shift the analysis a bit more explicitly towards the DLM’s properties and study them in more details. For example, when studying the masking rates, are the authors using full-canvas or block-wise decoding? Is it possible that beyond the masking rate, the total number of masks (which would on average be smaller in block-wise decoding) also matters? Do the results stay the same when we consider on-policy samples generated by the model using for example confidence-based sampling (rather than correct answers masked at different rates, which I believe the authors are using now)? Do we see any differences between Dream (finetuned from an AR model) and LLaDA (trained from scratch with the MDM loss)?
Questions for authors

See Weaknesses above.

4 comments

Thank you for your insightful and helpful suggestions, and we hope our point-by-point responses below address your concerns.

Response to W1. Our framework is designed to estimate and use head importance at different denoising stages. In our DLLM setting, the generation canvas has a predefined length, so the current masking rate is directly observable and can be used to select the corresponding head scores. Standard AR models extend a prefix without maintaining masked future positions, and their final output length is usually unknown during generation. Setting a maximum generation length does not provide an equivalent masking rate. Our masking-rate-conditioned attribution and dynamic allocation therefore cannot be transferred unchanged. Evaluating the full framework in AR models would require adapting the attribution objective and choosing a different variable to describe the generation stage. These choices would also need to be validated, since they could affect the results. Such an evaluation would assess an adapted framework with additional design choices. We therefore focus the present study on DLLMs, while leaving this extension for future work.

Response to W2. Decoding setup. We use block-wise decoding to generate reference completions and for downstream evaluation on LLaDA and Dream, following the settings of SparseD (Wang et al., ICLR 2026). We choose this setup for its better task performance. We have clarified this in Section 6.1 of the revised paper. Here, the masking rate is calculated using the full response length as the denominator.

Response to W2. Absolute mask count. We agree that the absolute number of masked tokens may also affect head importance. We use masking rate because it provides a common, length-normalized condition across responses. With limited attribution data, conditioning on absolute counts can give uneven coverage because large counts can only be evaluated on long responses. Using the same masking rates lets responses of different lengths contribute at every selected rate, making better use of the available attribution data.

Response to W2. On-policy attribution. We would like to clarify that "reference completion" refers to a response generated by the model itself. We randomly mask this response at different rates. Both the visible tokens and the reconstruction targets come from the generated response. We have clarified this in Section 3.1 of the revised paper. To examine actual decoding states, we ran ARA on cached intermediate states, preserving the generated tokens and mask positions from decoding. Both setups use generated tokens as targets. Results below use LLaDA and Table 1's pruning and evaluation settings (mean ± population SD over two runs). Lower scores after head removal indicate more influential selected heads.

SetupMMLUCMMLUC-EvalGPQA
ARA (real decoding states)47.24 ± 0.9241.83 ± 1.1641.60 ± 2.0125.50 ± 1.00
ARA (random masks, Table 1)44.54 ± 1.9135.09 ± 0.1732.91 ± 1.9325.25 ± 0.25

ARA with random masks gives lower mean scores on all four tasks. Random masking lets us reuse each generated response to create several attribution inputs at the same masking rate. Each mask leaves a different subset of tokens visible, allowing us to assess head contributions under different contexts. A single decoding trajectory usually provides only one state at a given rate. We average the attributions over these different states. We chose this design to reduce the need to collect additional generated responses and make attribution more practical with limited calibration data. We have added this explanation to Section 3.1 of the revised paper.

Response to W2. Differences between Dream and LLaDA. We observe differences in sensitivity to head removal and the stability of head rankings. Under our pruning settings, Dream produced incoherent outputs when more than about 10% of KV groups were removed, while LLaDA did so above about 40% of heads per layer. In Figure 2, the correlation between rates 0.15 and 0.90 is higher for LLaDA on HumanEval (0.67 vs. 0.45), but lower on Minerva Math (0.51 vs. 0.64). Figure 6 shows stronger cross-task agreement for LLaDA. The mean correlation within multiple-choice tasks is 0.82 for LLaDA and 0.49 for Dream, and the HumanEval-MBPP correlation is 0.85 versus 0.72. These results suggest that LLaDA has more consistent head-importance structure across tasks and tolerates more head removal under our tested settings.

References

Zeqing Wang, Gongfan Fang, Xinyin Ma, Xingyi Yang, and Xinchao Wang. SparseD: Sparse Attention for Diffusion Language Models. ICLR 2026.

Thank you for your response, I provide some additional comments below.

W1. While I agree with the authors that the specific conditioning method for the attribution scores would need to be adjusted to be directly applicable to the AR models, it seems to me that the ARA method itself (as described in Section 5.2) could still be applied to autoregressive models. I would encourage the authors to discuss in their paper the challenges and steps which would need to be taken to adapt their approach to autoregressive models.

W2. Decoding setup. Thank you for this clarification. Your response mentions that you use the full response length in the denominator. However, I dare arguing that during DLLM decoding the full response length is typically not known a priori, similarly as in the AR case (as the model can just generate EOS prematurely). As far as I understand, if the authors follow the setup of SparseD, then there the EOS generation is ignored, and indeed at all points the model has access to an entire fixed generation canvas. However, this approach is not very efficient when generating longer outputs, and some recent DLLM papers add tokens block by block instead (e.g. BD3-LM or even DiffusionGemma), finishing generation when an EOS token is generated. Hence, in those cases, I would assume that calculating the masking rate per block would be the most optimal and generalisable solution. Have the authors considered doing that? If not, I would encourage the authors to flag this as an important limitation of their work and the current decoding setup.

I am happy with the remaining answers and they address my other concerns.

Response to W1. Thank you for this helpful suggestion. We think ARA could be applied to AR models with appropriate extensions. We could retain the same head gates and asynchronous restoration paths and replace masked reconstruction with the log probability of each reference token given the prompt and preceding output tokens. Averaging contributions across output positions and responses would give a static head ranking. For dynamic allocation, it remains to be explored whether head importance varies with the number of tokens generated, as it does across denoising stages in DLLMs. Another challenge is preserving the data reuse offered by random masking. In our DLLM setup, one response provides multiple attribution inputs at the same masking rate. In standard AR prediction, the same reference response provides only one preceding context at a given output position. Obtaining diverse contexts at that position may therefore require additional generated responses. We will discuss these adaptations and challenges in the revised paper and explore this extension in future work.

Response to W2.decoding. Thank you for your detailed comments, which helped us better understand your concern. Our decoding setup follows SparseD and generates the response block by block within a fixed generation canvas. However, as you correctly noted, EOS does not terminate decoding early, so the entire canvas is generated. We agree that this leaves room for improving inference efficiency. For decoding that adds new blocks until EOS is reached, calculating the masking rate within the current block is a natural extension of our conditioning scheme. ARA (Global) also provides an alternative that does not require selecting scores by masking rate during inference. In our existing head removal experiments, its average scores across eight tasks are 31.85 on Dream and 33.31 on LLaDA (Table 2), lower than all baselines in Table 1, whose best averages are 36.08 and 35.31, respectively. Lower scores after removing the most important heads indicate that the selected heads have a greater effect on task performance. These results show that ARA can identify influential heads even when the same ranking is used throughout inference. Thank you for highlighting this valuable direction. We will clarify these points in the paper. In future work, we will explore using the masking rate within each block or a global ranking when the output length is not fixed.

Thank you, I am happy with these explanations and have increased my score.

Reviewer_eQRHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed ARA (Asynchronous Restoration Attribution), a path-based attribution method which independently samples restoration schedules for attention heads and investigates how attention-head importance varies across denoising stages in diffusion language models (DLLMs). Unlike standard Integrated Gradients (IG), which restores all components synchronously along a diagonal path, ARA evaluates head contributions under heterogeneous restoration configurations. The authors establish completeness and full-dimensional support properties for the proposed attribution rule. Experiments on LLaDA and Dream across different benchmarks demonstrate improvements in identifying influential attention heads through pruning and guiding adaptive sparse attention.

Critical strengths

S1. Interesting Motivation and Clear Illustration. Inspired by the Randomized Path Attribution proposed in Jeon, Jeong & Choi · ICCV 2023, this paper extends path-based attribution to attention heads in diffusion language models. The motivation is clearly presented, and the illustrative example effectively demonstrates how asynchronous restoration can capture interaction-dependent contributions that may be overlooked by conventional synchronized paths.

S2. Solid empirical evaluation . The main evaluation is conducted across diverse benchmarks, which is solid. Furthermore, the ablation studies are well designed and provide convincing empirical evidence for the effectiveness of the proposed approach. In particular, he comparison between synchronized and asynchronous restoration under matched gradient-evaluation budgets helps isolate the benefit of the proposed path design, while the comparison between dynamic and global head selection supports the importance of stage-dependent attribution.

Critical weaknesses

W1. Insufficient Comparison with existing randomized path attribution methods: Although reasonable to borrow thought from Jeon, Jeong & Choi · ICCV 2023, some other comparisons of the proposed restoration schedule over existing randomized path methods is not enough. For example, a comparison with SPI-style randomized paths under matched computational budgets would strengthen the methodological contribution.

W2. The Inconsistency between the proposed masking rate and the real denoising stage: The authors argued that “Attention Head Importance changes across denoising stages.” But the problem is that the context in a real DLLM denoising process typically includes tokens generated in previous steps of the model, rather than ground-truth tokens.

Questions for authors

Q1. Traditional Integrated Gradients (ARAs) can only observe the synchronization recovery state, thus potentially missing interactions between different heads. The authors prove, using a hand-constructed binary function, that this is mathematically possible. However, the question remains: How strong is this head interaction in DLLM? Is it truly the primary reason why ARAs outperform DPs?

Q2. ARA's random recovery path can produce very extreme head configurations. However, in the main pruning experiment of LLaDA, the authors only removed about 20% of the Heads per layer; Dream's removal budget is even smaller.

Q3. In Appendix A.3, I found the experiments were conducted by first calculating the importance based on the corresponding attribute data, then doing the dynamic pruning. But what if, for example, in real world scenarios, If the tasks are unseen or mixed, will Adaptive Sparse Attention still be effective?

2 comments

Thank you for your insightful and helpful suggestions, and we hope our point-by-point responses below address your concerns.

Response to W1. Thank you for pointing out this missing baseline. We added an SPI-style baseline with the same attribution data, masks, and 32 gate-gradient evaluations as ARA. SPI averages 36.40±0.07 on Dream and 35.70±0.25 on LLaDA, versus 30.52±0.78 and 31.64±0.70 for ARA. Lower post-removal scores indicate more influential selected heads. The full pruning comparison across both models and all eight tasks is reported in Table 1 of the revised paper, with implementation details in Appendix A.1. Uniform-base SPI equally weights a path and its reflection, which swaps late restoration with early completion. For some interactions that affect model behavior, these paired paths give opposite contributions that cancel in the exact path average. ARA starts heads at different times but fully restores all at the same endpoint, breaking this symmetry and potentially retaining these signals. In the smooth, monotone example in Appendix A.8, ARA matches the head ranking under both small strength reductions and complete suppression. SPI's mean ranks the most influential head last.

Response to W2. We apologize for the unclear wording. In Section 3.1, "reference completion" refers to a response generated by the model itself, not a ground-truth answer. We first generate a complete response and then randomly mask it at selected rates. Both the visible tokens and the reconstruction targets come from this generated response. Masking rate serves as a controlled proxy for denoising stage. We have clarified this in Section 3.1 of the revised paper.

Response to Q1. DP (Dynamic) and ARA (Dynamic) in Table 2 use the same data, masks, stage conditioning, and removal protocol, with 32 gate-gradient evaluations each. ARA lowers post-removal macro-average scores by 4.42 points on Dream and 3.96 on LLaDA, identifying heads with larger functional effects. For a gate objective additive across heads, exact integration gives identical DP and ARA attributions. Their key difference is how they assess head interactions. DP restores all heads together, while ARA evaluates each head with other heads restored to different degrees. This allows ARA to capture interaction-dependent contributions that the synchronized path can miss. The gains in Table 2 under matched settings support this explanation for ARA's advantage over DP.

Response to Q2. In our tests, LLaDA generated incoherent outputs when more than about 40% of heads per layer were removed. Dream collapsed above about 10% of KV groups. When most heads are off, restoring only a few heads is not enough to improve predictions of the reference tokens. ARA measures how each head changes the log-probability of these tokens as it is restored. If this score barely changes, that part of the path contributes little to the head's attribution. Thus, extreme configurations do not contribute much to the final attribution. The pruning experiment in our paper evaluates head rankings through their effect on task performance. We use 20% of heads per layer for LLaDA and 5% of KV groups for Dream to make performance differences between attribution methods easier to compare while staying below the observed collapse levels. Removing too few heads can leave little difference in task scores, whereas removing too many can make all methods produce incoherent outputs. The chosen budgets are around half the observed collapse levels, and all methods use the same budget within each model.

Response to Q3. We compared task-specific and mixed attribution for adaptive sparse attention on MMLU and GSM8K. Task-specific ARA uses attribution data from each target task itself. To simulate unseen tasks, Mixed ARA excludes both MMLU and GSM8K and averages attribution scores from the other six benchmarks. The same mixed scores are used for both target tasks. We then compared their downstream performance on LLaDA at a 30% retained-block ratio.

SetupMMLUGSM8K
Dense65.0980.00
Task-specific ARA64.1279.75
Mixed ARA63.7778.25

Mixed ARA preserves much of the dense model's performance. Relative to task-specific ARA, scores drop by 0.35 points on MMLU and 1.50 on GSM8K. This is consistent with MMLU's higher mean pairwise Spearman correlation with the six other tasks (0.598 for MMLU vs. 0.315 for GSM8K). We have added this experiment and analysis to Appendix A.3 of the revised paper.These results suggest that mixed calibration is a possible option when the task is unknown, though transfer quality varies. Our main study focuses on known downstream tasks with available attribution data. We will study unseen and mixed-task settings more broadly in future work.

Thank you again for your helpful suggestions. If our replies address your concerns, we would be grateful if you could reconsider your score. We would be happy to discuss further.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors’ efforts in addressing my concerns.

The new SPI baseline, clarification of the masking procedure, and mixed-task experiments have adequately addressed my major concerns and strengthened the paper.

I am therefore happy to raise my rating from 2 to 3.

Comments on the paper

2 comments

We thank all the reviewers for their insightful and encouraging feedback. Their comments helped us improve the clarity and evaluation of our work. In the revised paper, we added an SPI-style randomized-path baseline under the same attribution budget (Table 1) and a third seed for all variants on both models in Table 2. ARA retains its advantage in the three-seed comparison. We clarified that both the visible tokens and targets come from model-generated responses, explained why random masking makes better use of limited attribution data (Section 3.1), and specified the block-wise decoding setup (Section 6.1). We also added an example showing how ARA preserves interaction signals that SPI can cancel (Appendix A.8), a mixed-task calibration experiment on held-out tasks (Appendix A.3), and attribution timings with an explanation of score reuse during inference (Appendix A.1). New material in the paper is marked in blue.

Our responses below also report experiments using real decoding states, pruning results on LLaDA2.0-16B, per-path variance, and a second seed for sparse attention at 30% retention. These results provide further evidence on attribution quality, scalability, and stability. We hope the revisions and additional results address your concerns, and we would be happy to discuss any remaining questions.

Dear reviewers,

A reminder that the discussion period will end on November 18 at 11:59pm AoE. I encourage you to interact with the authors' rebuttal to clarify any remaining issue (and how well the rebuttal addressed your concerns) before the next phase.

Thanks, The AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.