acceptodds
Under review as a conference paper at ICLR 2027

UaRA: Uncertainty-aware Rank Allocation for Parameter-Efficient Fine-Tuning of Diffusion Large Language Models

Abstract

Diffusion large language models (dLLMs) have recently attracted increasing attention for their iterative denoising paradigm and potential for flexible parallel generation. Unlike autoregressive generation, diffusion decoding operates over continuously changing noise states, leading to varying adaptation demands throughout the generation process. However, existing parameter-efficient fine-tuning (PEFT) methods such as LoRA are primarily designed for autoregressive models and do not account for the dynamic adaptation demands of diffusion decoding. In particular, standard LoRA adopts a fixed rank, allocating the same adaptation capacity to all tokens and noise conditions despite their varying prediction difficulty. Existing adaptive-rank methods such as AdaLoRA are also not directly suited to dLLMs, because their gradient-based importance estimates can be distorted by noise-dependent loss scaling rather than reflecting intrinsic parameter importance. To address this issue, we propose UaRA: Uncertainty-aware Rank Allocation for Parameter-Efficient Fine-Tuning of Diffusion Large Language Models, which dynamically allocates low-rank capacity according to the evolving adaptation demand across noise levels. UaRA first calibrates gradient-weight sensitivity using the known loss-scaling coefficient, removing noise-induced scale bias from parameter-importance estimation. It then models the importance of each rank component as a function of the noise level using second-order Legendre bases fitted by online recursive least squares, enabling the active-rank distribution to adapt continuously throughout diffusion training. Extensive experiments demonstrate that UaRA consistently outperforms strong PEFT baselines, validating the effectiveness of uncertainty-aware and noise-conditioned rank allocation for dLLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion Large Language Models,Parameter-Efficient Fine-Tuning,LoRA

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ryHjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes UaRA, a noise-conditioned adaptive-rank PEFT framework for diffusion large language models (dLLMs). UaRA first calibrates gradient-weight sensitivity using the known loss-scaling coefficient. It then uses second-order Legendre basis functions and online recursive least squares (RLS) with a forgetting factor to learn a noise-dependent importance function for each rank component. By incorporating the exponential moving average (EMA) of prediction residuals, UaRA selects active rank components under a fixed global budget. Experiments on eight commonsense reasoning and four mathematical reasoning benchmarks using LLaDA-8B-Base/Instruct show that UaRA achieves higher average accuracy than the baselines.

Critical strengths

The paper proposes an uncertainty-aware and noise-conditioned adaptive-rank framework for dLLMs.

The framework combines scale-calibrated component sensitivity, noise-conditioned importance estimation, and budgeted rank allocation to dynamically select low-rank components across different diffusion states.

Critical weaknesses

The paper's motivation assumes that different noise states require different adaptation ranks. Although UaRA outperforms fixed-rank methods in some experiments, the performance gains are relatively limited. Therefore, the necessity of dynamic rank allocation is not sufficiently supported by experimental evidence.

On commonsense reasoning benchmarks, UaRA improves average accuracy over the strongest baselines by only approximately 0.25–0.26 percentage points. On mathematical reasoning with LLaDA-8B-Instruct, UaRA achieves an average accuracy of 78.55%, only 0.05 percentage points higher than HiRA rank-48 (78.50%), and does not outperform the baselines on all individual tasks. LoRA rank-32 still performs better on some tasks, such as HellaSwag and PIQA.

Questions for authors

Please conduct experiments to verify the necessity of dynamically adjusting ranks across different noise states.

Please provide the mean and variance over multiple runs or conduct significance tests, and discuss more explicitly which tasks show limited performance gains and the reasons behind them.

1 comment

We sincerely thank the reviewer for highlighting UaRA’s uncertainty-aware design and its use of scale-calibrated sensitivity to allocate ranks across diffusion states.

Question 1: Evidence for Noise-Conditioned Rank Allocation

Answer: We appreciate the suggestion to test directly whether varying the allocation improves accuracy. On LLaDA-8B-Base, UaRA with an average active rank of 32 achieves 76.77% mean math accuracy, compared with 75.64% for the fixed-rank variant (+1.13%, Appendix C.2, Figure 7). With the active budget fixed, 89 of 128 modules change rank between t = 0.2 and t = 0.4, and about 33% of the selected components change between the sampled low- and high-noise levels (Appendix C.7, Figure 10). Moreover, UaRA outperforms LoRA-48 in 30 of 36 comparisons across the original and additional benchmarks on the two backbones. On the original Base math suite, UaRA improves the mean by +1.93% over LoRA-32 (76.77% vs 74.84%) and GSM8K accuracy by +6.06% over NaRA-32 (75.58% vs 69.52%).

To clarify the advantages of our method, we conducted experiments on additional benchmarks (Appendix C.6, Tables 9–10). UaRA achieves the highest accuracy on all six added benchmarks with both backbones. On Instruct, the largest task-level gains in the two groups are on MMLU (+2.70%) and SVAMP (+2.00%).

Additional benchmark results

Base ModelMethodCSQAMMLUCOPAWSC273Avg.
LLaDA-8B-BaseZero-shot79.0361.4793.2071.7976.37
LLaDA-8B-BaseLoRA-4884.9366.8796.6091.5885.00
LLaDA-8B-BaseNaRA-4885.2666.4497.0094.1485.71
LLaDA-8B-BaseHiRA-4881.7466.6296.2083.5282.02
LLaDA-8B-BaseUaRA85.5067.2598.0095.0486.45
LLaDA-8B-InstructZero-shot82.3163.6995.4084.2581.41
LLaDA-8B-InstructLoRA-4883.8763.7497.2091.2184.01
LLaDA-8B-InstructNaRA-4883.9564.2097.0091.2184.09
LLaDA-8B-InstructHiRA-4882.4764.2496.2086.4582.34
LLaDA-8B-InstructUaRA84.6866.9498.0092.3185.48
Base ModelMethodMATH-500SVAMPAvg.
LLaDA-8B-BaseZero-shot24.6063.7044.15
LLaDA-8B-BaseLoRA-4824.6059.6042.10
LLaDA-8B-BaseNaRA-4830.6079.8055.20
LLaDA-8B-BaseHiRA-4819.0048.8033.90
LLaDA-8B-BaseUaRA31.6081.2056.40
LLaDA-8B-InstructZero-shot33.8080.9057.35
LLaDA-8B-InstructLoRA-4832.8083.7058.25
LLaDA-8B-InstructNaRA-4833.8083.5058.65
LLaDA-8B-InstructHiRA-4834.2087.6060.90
LLaDA-8B-InstructUaRA35.6089.6062.60

Question 2: Statistical Significance and Task-Level Variation

Answer: Thanks for asking us to quantify the uncertainty and clarify where the gains are limited. In a separate analysis, we paired saved predictions on the same questions across GSM8K, AddSub, AQuA, and MultiArith and applied Holm correction across 15 comparisons. The tests quantify question-sampling uncertainty for the fixed checkpoints under their original inference settings.

BaselineGSM8K gain (adjusted p)Four-task mean gain (adjusted p)
LoRA+3.49% (0.027)+1.96% (0.023)
HiRA+15.92% (<0.001)+9.08% (0.0013)
NaRA+2.88% (0.011)+1.34% (0.048)

UaRA also improves MultiArith accuracy over HiRA by +15.00% (adjusted p < 0.001).

On LLaDA-8B-Base, LoRA-32 leads on HellaSwag (95.40% vs 94.65%) and PIQA (88.08% vs 87.16%). Across the remaining six benchmarks, UaRA averages 86.38% versus 85.66% for LoRA-32. On LLaDA-8B-Instruct, HiRA-48’s +2.79% lead on AddSub (86.33% vs 83.54%) offsets most of UaRA’s gains on the other three math tasks. Across GSM8K, AQuA, and MultiArith, UaRA averages 76.88% versus 75.89% for HiRA-48 (+0.99%). Across the six additional benchmarks (Appendix C.6, Tables 9–10), UaRA improves mean accuracy over NaRA-48, the strongest baseline on both backbones, by +0.89% on Base (76.43% vs 75.54%) and +2.25% on Instruct (77.86% vs 75.61%).

Our variability results currently cover GSM8K on LLaDA-8B-Base. We will extend this analysis to the remaining benchmarks.

MethodGSM8K accuracy (%)
Zero-shot67.02 ± 0.07
LoRA-3269.18 ± 0.09
LoRA-4868.13 ± 0.11
NaRA-3269.49 ± 0.06
NaRA-4868.86 ± 0.03
HiRA-3241.40 ± 0.12
HiRA-4855.90 ± 0.10
AdaLoRA49.90 ± 0.12
UaRA75.58 ± 0.11
Reviewer_wh8pOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper tackles a gap in rank allocation during LoRA fine-tuning on dLLMs (Duffision Language Models). The authors argue that existing LoRA rank-allocation mechanisms are suboptimal at dLLMs because they treat every step of this process the same way, under a fixed amount of resources. In reality, the model's needs change depending on the amount of noise in different stages. The proposed method introduces a dynamic system that senses the current noise level and prediction uncertainty, thereby reallocating learning capacity. Primarily, it allows the model to focus its rank allocation where and when it is most needed during generation.

Critical strengths
  1. A combination of dLLMs and LoRA rank-allocation into one work.
  2. The authors introduce a calibration step that ensures parameter importance is judged on intrinsic merit rather than noise-induced magnitude.
  3. The method consistently outperforms standard LoRA and recent dLLM-specific baselines across various tasks.
Critical weaknesses
  1. Compared to the simplicity of standard LoRA, UaRA introduces several moving parts, which may make it harder for practitioners to adopt without specialized codebases.
  2. While the method optimizes the "active" rank, it requires maintaining a larger set of "candidate" parameters and performing online estimation.
  3. The effectiveness of the method might be tightly coupled with the specific noise schedule used during training.
Questions for authors
  1. Could you compare the computational cost with the standard fixed-resource approach?
  2. Since this work is primarily focused on dLLMs, will conventional visual diffusion models benefit from your approach?
  3. For a practitioner who wants a simple solution, how sensitive is this method to its own internal settings or hyperparameters?
3 comments

We sincerely appreciate the reviewer for recognizing the value of adaptive rank allocation for diffusion LLMs and the role of scale calibration in estimating component importance.

Question 1: Computational Cost

Answer: We appreciate this constructive suggestion, and we benchmark the rank-32 baselines against UaRA with a candidate rank of 48 and an average active rank of 32 (Appendix C.5, Table 8).

MethodTraining step (ms)Training peak VRAM (GB)Inference step (ms)Inference peak VRAM (GB)
LoRA-32372.3816.484110.9715.309
HiRA-32460.2316.484140.2815.309
NaRA-32409.6316.485112.0315.310
UaRA-48/32470.5416.892122.1515.686

Compared with LoRA-32, UaRA takes 26.4% longer per training step and 10.1% longer per inference step, using 0.408 GB and 0.377 GB more peak VRAM, respectively. Its inference step time is 12.9% lower than HiRA-32's. We will open-source the UaRA implementation to support adoption and reproducibility.

Question 2: Applicability to Visual Diffusion Models

Answer: We thank the reviewer for raising this insightful concern. In our SDXL DreamBooth experiment on the dog6 subject, UaRA achieves the highest DINO and CLIP-I scores and the lowest FID. Appendix C.8 (Figure 11) shows UaRA-generated examples of the same subject in studio and forest settings.

MethodDINO ↑CLIP-I ↑FID ↓
LoRA0.85990.896868.38
NaRA0.84360.914467.91
UaRA0.87150.924160.20

Question 3: Hyperparameter Sensitivity

Answer: We tested four minimum active ranks, three RLS forgetting factors, and three cross-noise aggregation schemes (Appendix C.2–C.3).

  • Minimum active rank (Figure 7). Mean math accuracy ranged from 75.58% to 76.77%, peaking at a minimum rank of 24. At an average active rank of 32, this floor leaves 25% of the budget for redistribution across modules.
  • RLS forgetting factor (Table 7). This controls how quickly past importance observations are discounted. Under uniform aggregation weights, 0.98 gave the highest mean math accuracy (75.17%), compared with 74.90% at 0.97 and 73.79% at 0.99.

Across the three aggregation schemes, mean commonsense accuracy ranged from 87.19% to 87.51% under the tested training noise schedule (Table 6). Table 5 (Appendix B.2) lists the main Math14k configuration. If you have any other questions, please let us know.

Thanks for the detailed response. Q1 to Q3 are answered, but the dependence on the training noise schedule (W3) remains untested. Please add the results if available, or state it as a limitation.

We thank the reviewer for raising the concern about the potential impact of the training noise schedule. For example, all compared methods, including LoRA, HiRA, NaRA, and UaRA, use the same uniform noise sampling and random token masking protocol.

Specifically, for each training example, we keep the question prompt unchanged and independently sample a noise level t∼U(0,1)t \sim \mathcal{U}(0, 1). Given LL tokens in the supervised answer span, we uniformly sample k=⌈tL⌉k = \lceil tL \rceil positions without replacement and replace their tokens with [MASK], ensuring that at least one token is masked. The model predicts the original tokens at these positions, conditioned on the question prompt and the remaining visible answer tokens. The training loss is computed only at masked positions, with inverse-noise weighting. Both the noise level and the masking positions are resampled each time an example is processed.

Thus, all methods encounter examples spanning low, medium, and high noise levels throughout training. We use neither a noise curriculum that changes with training progress nor a noise distribution tailored to UaRA. This ensures that the methods are compared under the same training noise protocol.

We are grateful to the reviewer for highlighting this important point. We will incorporate this clarification into the revised paper and explicitly discuss the potential dependence on alternative training noise schedules as a limitation and direction for future work.

Reviewer_KDSEOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UaRA, a rank allocation method for fine-tuning diffusion LLMs. It adjusts LoRA ranks based on the current noise level, using calibrated gradient sensitivity and online regression to estimate component importance.

Experiments on two LLaDA-8B variants across commonsense and math benchmarks show improvements over several PEFT baselines. Ablations also support the proposed importance estimation and calibration methods.

Critical strengths
  1. Different diffusion states may need different adaptation capacities, so allocating ranks based on noise levels makes sense. I like that the method addresses a limitation of fixed-rank LoRA rather than simply adding more parameters.

  2. Tables 3 and 4 show meaningful gains from noise-conditioned importance estimation and scale calibration. The rank allocation visualizations also help explain what the method is doing.

Critical weaknesses
  1. In Table 1, UaRA beats the strongest baseline by only 0.25 – 0.26 percentage points. On mathematical reasoning with LLaDA-8B-Instruct, the advantage is just 0.05. Even the paper’s own numbers suggest that run-to-run variation may exceed these gains. Appendix Table 5 specifies an RLS forgetting factor of 0.99, yet Table 7 reports 73.79% average accuracy for that setting (and at most 75.17% for any tested value), compared with 76.77% in the main results. Similarly, the residual-weighting ablation reports a three-run mean of 92.26% on MultiArith with LLaDA-8B-Instruct, while Table 2 reports 96.83% for the same backbone. These may use different configurations, but the differences need clarification.

  2. Figure 4(a) shows that RLS predictions are zero near t=0.3t = 0.3 and t=0.5t = 0.5. Since predictions are clipped at zero (Eq. 9), a zero mean over a projection type implies that every component’s prediction is zero there. Figure 8 reports the same at t=0.1t = 0.1. At these noise levels all scores are zero, and Eq. (11) selects components by the fixed tie-breaking order rather than by importance. The nearly uniform allocations at t=0.2–0.4t = 0.2 – 0.4 in Figures 5 and 9 are consistent with this. Over a large part of the noise range, the allocation therefore does not appear to be noise-conditioned in the intended sense.

Questions for authors

Please see the Weakness

1 comment

We sincerely thank the reviewer's valuable suggestions.

Question 1: Modest gains and differences in experimental settings

Answer: We appreciate the reviewer’s observation. Although UaRA's gains vary by task, it beats LoRA-48 on 30 of 36 benchmarks (main and additional) across LLaDA-8B-Base and LLaDA-8B-Instruct at an identical 0.624% parameter budget. On Base, UaRA improves mean math accuracy +1.93% over LoRA-32 (76.77% vs 74.84%) and GSM8K accuracy by +6.06% over NaRA-32 (75.58% vs 69.52%). For math reasoning tasks on Instruct, UaRA and HiRA-48 are nearly tied on the four-task math average (78.55% vs. 78.50%). This is because HiRA-48's high gains (+2.79%) on AddSub (86.33% vs. 83.54%). If we compare it with UaRA on the other three benchmarks, the mean result becomes 76.88% for UaRA vs 75.89% for HiRA-48 (+0.99%).

To clarify the advantages of our method, we conducted experiments on additional benchmarks (Appendix C.6, Tables 9–10). UaRA gets the highest scores on all six added benchmarks with both backbones. Mean gains over the strongest baseline reach +1.39% on commonsense reasoning and +1.70% on math reasoning. The tables report accuracy (%) and unweighted means. All tuned methods use 0.624% trainable parameters. UaRA uses candidate, minimum, and average active ranks of 48, 28, and 32, respectively.

Additional benchmark results

Base ModelMethodCSQAMMLUCOPAWSC273Avg.
BaseZero-shot79.0361.4793.2071.7976.37
BaseLoRA-4884.9366.8796.6091.5885.00
BaseNaRA-4885.2666.4497.0094.1485.71
BaseHiRA-4881.7466.6296.2083.5282.02
BaseUaRA85.5067.2598.0095.0486.45
InstructZero-shot82.3163.6995.4084.2581.41
InstructLoRA-4883.8763.7497.2091.2184.01
InstructNaRA-4883.9564.2097.0091.2184.09
InstructHiRA-4882.4764.2496.2086.4582.34
InstructUaRA84.6866.9498.0092.3185.48

Table 10. Additional math benchmarks

Base ModelMethodMATH-500SVAMPAvg.
BaseZero-shot24.6063.7044.15
BaseLoRA-4824.6059.6042.10
BaseNaRA-4830.6079.8055.20
BaseHiRA-4819.0048.8033.90
BaseUaRA31.6081.2056.40
InstructZero-shot33.8080.9057.35
InstructLoRA-4832.8083.7058.25
InstructNaRA-4833.8083.5058.65
InstructHiRA-4834.2087.6060.90
InstructUaRA35.6089.6062.60

Configuration differences. The main results math run use high weights (0.05, 0.10, 0.15, 0.30, 0.40) and a minimum rank of 24 (76.77%). At an RLS forgetting factor of 0.99, the Table 7 ablation uses uniform cross-noise weights (0.20, 0.20, 0.20, 0.20, 0.20) and a minimum active rank of 24 (73.79% mean accuracy). The Instruct residual-weighting ablation uses high weights (0.05, 0.10, 0.15, 0.30, 0.40) and a minimum rank of 28 (92.26% on MultiArith). We highlight the different rows in Section 4.3, Appendix C.3, Appendix B.2, and the additional benchmark discussion in Appendix C.6 in red.

Question 2: Do zero scores make rank allocation fixed?

Answer: We appreciate this constructive suggestion, and we supplement the analysis results for this issue in Appendix C.7. To examine the allocation directly, we sweep the noise level and inspect the final scores and selected ranks.

  • Final scores (Figure 10(a)). At t = 0.3, where the zero-score fraction peaks, 81 ranks still score above zero. At t = 0.5, 2,200 ranks score above zero.
  • Selected ranks (Figure 10(b)). The selector adaptively reconfigures the active rank set across noise regimes: the Jaccard similarity decreases from 0.957(t=0.2 vs 0.4) to 0.695 (t=0.3 vs 0.5), and further falls to 0.488 - 0.498 when comparing low-noise (0.2, 0.3, 0.4) against high-noise (0.7, 0.9), indicating that roughly 33% of the budget is dynamically reallocated to distinct subspaces.
  • Per-module ranks (Figure 10(c)). Between t = 0.2 and 0.4, 89 of 128 modules (69.5%) change active ranks.

Jaccard similarity between selected rank sets

Noise level0.20.30.40.50.70.9
0.21.0000.9840.9570.7020.4950.495
0.30.9841.0000.9470.6950.4980.498
0.40.9570.9471.0000.7240.4890.488
0.50.7020.6950.7241.0000.4220.420
0.70.4950.4980.4890.4221.0000.988
0.90.4950.4980.4880.4200.9881.000

We added Figure 10 in Appendix C.7 and marked the accompanying analysis in red. If you have any other concerns, please let us know.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.