- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes UaRA, a noise-conditioned adaptive-rank PEFT framework for diffusion large language models (dLLMs). UaRA first calibrates gradient-weight sensitivity using the known loss-scaling coefficient. It then uses second-order Legendre basis functions and online recursive least squares (RLS) with a forgetting factor to learn a noise-dependent importance function for each rank component. By incorporating the exponential moving average (EMA) of prediction residuals, UaRA selects active rank components under a fixed global budget. Experiments on eight commonsense reasoning and four mathematical reasoning benchmarks using LLaDA-8B-Base/Instruct show that UaRA achieves higher average accuracy than the baselines.
Critical strengths
The paper proposes an uncertainty-aware and noise-conditioned adaptive-rank framework for dLLMs.
The framework combines scale-calibrated component sensitivity, noise-conditioned importance estimation, and budgeted rank allocation to dynamically select low-rank components across different diffusion states.
Critical weaknesses
The paper's motivation assumes that different noise states require different adaptation ranks. Although UaRA outperforms fixed-rank methods in some experiments, the performance gains are relatively limited. Therefore, the necessity of dynamic rank allocation is not sufficiently supported by experimental evidence.
On commonsense reasoning benchmarks, UaRA improves average accuracy over the strongest baselines by only approximately 0.25–0.26 percentage points. On mathematical reasoning with LLaDA-8B-Instruct, UaRA achieves an average accuracy of 78.55%, only 0.05 percentage points higher than HiRA rank-48 (78.50%), and does not outperform the baselines on all individual tasks. LoRA rank-32 still performs better on some tasks, such as HellaSwag and PIQA.
Questions for authors
Please conduct experiments to verify the necessity of dynamically adjusting ranks across different noise states.
Please provide the mean and variance over multiple runs or conduct significance tests, and discuss more explicitly which tasks show limited performance gains and the reasons behind them.
We sincerely thank the reviewer for highlighting UaRA’s uncertainty-aware design and its use of scale-calibrated sensitivity to allocate ranks across diffusion states.
Question 1: Evidence for Noise-Conditioned Rank Allocation
Answer: We appreciate the suggestion to test directly whether varying the allocation improves accuracy. On LLaDA-8B-Base, UaRA with an average active rank of 32 achieves 76.77% mean math accuracy, compared with 75.64% for the fixed-rank variant (+1.13%, Appendix C.2, Figure 7). With the active budget fixed, 89 of 128 modules change rank between t = 0.2 and t = 0.4, and about 33% of the selected components change between the sampled low- and high-noise levels (Appendix C.7, Figure 10). Moreover, UaRA outperforms LoRA-48 in 30 of 36 comparisons across the original and additional benchmarks on the two backbones. On the original Base math suite, UaRA improves the mean by +1.93% over LoRA-32 (76.77% vs 74.84%) and GSM8K accuracy by +6.06% over NaRA-32 (75.58% vs 69.52%).
To clarify the advantages of our method, we conducted experiments on additional benchmarks (Appendix C.6, Tables 9–10). UaRA achieves the highest accuracy on all six added benchmarks with both backbones. On Instruct, the largest task-level gains in the two groups are on MMLU (+2.70%) and SVAMP (+2.00%).
Additional benchmark results
Question 2: Statistical Significance and Task-Level Variation
Answer: Thanks for asking us to quantify the uncertainty and clarify where the gains are limited. In a separate analysis, we paired saved predictions on the same questions across GSM8K, AddSub, AQuA, and MultiArith and applied Holm correction across 15 comparisons. The tests quantify question-sampling uncertainty for the fixed checkpoints under their original inference settings.
UaRA also improves MultiArith accuracy over HiRA by +15.00% (adjusted p < 0.001).
On LLaDA-8B-Base, LoRA-32 leads on HellaSwag (95.40% vs 94.65%) and PIQA (88.08% vs 87.16%). Across the remaining six benchmarks, UaRA averages 86.38% versus 85.66% for LoRA-32. On LLaDA-8B-Instruct, HiRA-48’s +2.79% lead on AddSub (86.33% vs 83.54%) offsets most of UaRA’s gains on the other three math tasks. Across GSM8K, AQuA, and MultiArith, UaRA averages 76.88% versus 75.89% for HiRA-48 (+0.99%). Across the six additional benchmarks (Appendix C.6, Tables 9–10), UaRA improves mean accuracy over NaRA-48, the strongest baseline on both backbones, by +0.89% on Base (76.43% vs 75.54%) and +2.25% on Instruct (77.86% vs 75.61%).
Our variability results currently cover GSM8K on LLaDA-8B-Base. We will extend this analysis to the remaining benchmarks.