- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper presents an analytical and experimental study that takes a step back to ask a practical question about adaptive LoRA methods: when rank gets shuffled around between layers, does it actually help because the capacity is placed in a smarter spot, or is it just accidentally throwing more parameters and training compute at the model? Specifically, the paper reveals two key factors: signal reliability and resource accounting. On the resource side, it points out that matching "total rank" is a trap because one unit of rank in a feed-forward layer costs way more parameters than a unit in an attention layer, meaning adaptive methods often sneakily activate 16% more parameters than uniform baselines. On the signal side, it builds a learning-progress score and derives a clean Spearman rank correlation identity showing that whenever global calibration loss randomly ticks up instead of down, the sign flip completely reverses the module rankings, even though the underlying gradient magnitudes stay rock-steady. By testing controlled rank transfers from a shared checkpoint and running synthetic toy models, the paper shows that immediate loss drops from changing LoRA scaling factors can masquerade as better learning, and that online reallocation really only shines when task demands actively shift mid-training.
Critical strengths
-
The paper identifies an interesting question that seems to be practical and important.
-
A lot of papers in the adaptive LoRA space just match the terminal total rank and call it a fair fight. This work does a great job pointing out that one rank unit in an attention projection costs way fewer parameters than one in a feed-forward projection.
-
The synthetic setups in Section 5 do a nice job illustrating when adaptive allocation actually matters in theory.
Critical weaknesses
-
The presentation of the entire paper feels pretty disjointed and jumps into heavy concepts without enough setup. I will report two examples here and there are many other similar inconsistencies in the paper. In Section 1, the third paragraph suddenly brings up "resource accounting" and dives straight into exact parameter counts for DeBERTa and training trajectories without properly motivating why the community has been getting this wrong or giving a smooth conceptual transition from the previous paragraph on signal reliability. Similarly, in Section 3.2, the term "calibration loss" is dropped right into the text out of nowhere. There is no clear upfront explanation in the main text of what this calibration loss actually is, why a separate calibration subset is needed instead of just using the training batch loss, or what its exact functionality is, leaving the reader hanging until tiny implementation details show up way back in Appendix B. Therefore, I feel that the paper is very hard to follow and I can’t get the main point of the paper.
-
There is a big disconnect between the real-task online score and the synthetic online score. In the main GLUE and Llama experiments, the online policy uses the learning-progress score (Equation 5), which distributes global loss changes across modules using current gradient norms. And a huge chunk of the paper is spent showing that this exact score suffers from sign flips. But then, when moving to the synthetic controls in Section 5 to show that online allocation actually helps under shifting demand, the paper ditches and swaps in a totally different "direct module probe" that perturbs one module at a time. This makes the narrative feel a bit disconnected: if the main online score has this huge sign-flip flaw, testing a completely different (and much more expensive) per-module probing score in the synthetic benchmark doesn't really show whether the main online method can actually track demand shifts.
-
The main experiments don't actually enforce the paper's own fair-accounting rule. Even though the paper makes a really strong case that matching total rank () is unfair because it changes by up to 16.4%, the main benchmark tables (Tables 2 and 4) still only match total rank rather than matching active parameter budgets . Aside from the small 20-step intervention on three module pairs in RTE and the GoRA component study (which accidentally uses fewer parameters due to rounding/clipping), there isn't a full end-to-end training baseline where uniform, one-shot, and online policies are strictly constrained to the exact same budget.
Questions for authors
-
Why not fix the sign-flip issue in and test a corrected online policy? Since Equation 6 shows that the negative Spearman correlation comes entirely from the shared global loss-difference sign integrating into the EMA , what happens if the online allocator just uses the magnitude or drops the factor altogether? Does training with a sign-corrected online score actually beat uniform allocation on GLUE and Commonsense?
-
How does the real-task learning-progress score () perform on the synthetic shifting-demand benchmark? In Section 5, the synthetic control swaps out for a direct per-module perturbation probe (which requires forward passes per event). If the standard online score from Section 3.2 is run on the synthetic phase-shift benchmark instead of the direct probe, does it still manage to track the demand shift at step 800, or do the sign flips wreck its ability to reallocate rank?