- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper investigates whether low-rank parameter-efficient fine-tuning benefits from postponing the point at which the final deployment capacity becomes binding. It introduces Delayed Capacity Commitment, and the remaining surplus is removed only at the end of training to satisfy the deployment budget.
The proposed implementation, Adaptive Competition Policy, represents DoRA-style directional updates as globally competing rank-one slots. It combines importance scores derived from parameter magnitudes and optimizer statistics with magnitude and trend terms. A controller uses competition-state statistics and development-set feedback to govern compression. An important implementation detail is that both the surviving set and its selection ranking freeze when the intermediate floor is reached; final truncation uses that saved ranking.
The primary contribution is therefore an empirical investigation of a particular buffered capacity trajectory, together with a framework for describing and comparing such protocols.
Critical strengths
-
Well-motivated research question. The paper identifies an interesting and practically relevant question in PEFT.
-
Consistent improvements in the primary setting. The proposed approach achieves positive capacity dividends across all five paired seeds on Qwen3-0.6B GSM8K at both tested buffer ratios.
Critical weaknesses
- Limited novelty compared with existing adaptive-rank methods.
The idea of delaying deployment-budget commitment is related to existing adaptive-rank methods, such as AdaLoRA and SoRA. Although DCC introduces an intermediate compression floor, the difference from existing compression schedules is not sufficiently discussed. Moreover, the paper lacks a fair comparison with AdaLoRA under the same experimental settings, making it difficult to evaluate the actual improvement over existing methods.
- Insufficient ablation studies on ACP.
ACP contains several components, including importance scoring, global rank-one competition, and an adaptive compression controller. However, the paper does not provide sufficient ablation studies to analyze the effectiveness of each component. In addition, Tab. 4 shows that the simpler fixed rank-24 training with terminal selection can achieve comparable performance to buffered ACP. Therefore, it is unclear whether all these components are necessary.
- Limited generalization across different tasks and settings.
The performance improvements are mainly observed on GSM8K. As shown in Tab. 5, the improvements on BoolQ, PiQA, MAWPS, and AQuA-RAT are marginal, negative, or inconclusive. Moreover, a larger buffer ratio of 0.3 does not bring an average improvement on Llama-3.1-8B. These results indicate that the effectiveness of the proposed method is sensitive to different tasks and hyperparameter settings. More analysis is needed to explain under what conditions the proposed method can achieve consistent improvements.
Questions for authors
-
What does the retained-buffer trajectory add beyond an established compression schedule?
-
Does delaying buffer removal help when the preceding history and final mask are identical?
3 comments
Dear Reviewer,
As we approach the final hours of the author-reviewer discussion period, I would greatly appreciate it if you could take a moment to review and respond to the authors’ rebuttal.
Please briefly indicate which concerns have been addressed and which remain, and consider updating your scores if needed.
Thank you again for your time and valuable contributions!
Best regards, AC
Response to the rebuttal
Thank you for the detailed rebuttal and additional experiments. The new controlled experiment helps clarify the effect of delayed commitment under identical training histories and final masks. However, I still have concerns regarding the limited novelty over existing adaptive-rank methods, insufficient validation of ACP's individual components, and inconsistent generalization across tasks and models. I appreciate the authors' efforts but will maintain my original rating.
Thank you for the two questions. We clarify the distinction between allocation and commitment below, and report the new experiment that directly holds the preceding history and final mask fixed.
Q1. What does the retained-buffer trajectory add beyond an established compression schedule?
ACP draws on DoRA's magnitude/direction parameterization and importance-based adaptive rank allocation. Its state-driven controller supplies a concrete implementation in which to investigate DCC. The research question is whether optimization should be restricted to the deployment budget as soon as adaptive allocation ends.
AdaLoRA's standard schedule reduces capacity to the target budget and then trains with the resulting allocation fixed (§3.3). Buffered ACP instead stops adaptive compression above that budget. For Qwen3-0.6B, the trajectory is 6,272 initial slots, 1,809 slots at the intermediate floor, and 1,568 deployment slots. The 241 surplus slots remain active and their weights continue updating until commitment.
An important implementation detail is that both the surviving set and the ranking used for final selection freeze at the intermediate floor. The deployment subset is therefore already identifiable then. DCC tests whether retaining the additional active capacity during subsequent optimization can still affect the final compact adapter. This does not require continued re-ranking after the floor.
A prescribed schedule can express the same trajectory. The contribution is the explicit separation and empirical comparison of allocation freeze and deployment-budget enforcement, rather than a claim that existing scheduling machinery cannot implement it. ACP is one realization of this comparison; the commitment question does not require its particular controller to be optimal.
The original within-ACP Qwen GSM8K comparisons give mean gains of +1.60 and +2.50 percentage points at the two buffer ratios, positive in all five paired seeds at each ratio. Those full-trajectory comparisons allow the final masks to differ. Your second question calls for a more specific control, which we have now completed.
Q2. Does delaying buffer removal help when the preceding history and final mask are identical?
In the new five-seed Qwen3-0.6B/GSM8K experiment, each run branches at its intermediate floor. One continuation removes the surplus immediately; the other retains it until training ends. They share the preceding training history and deploy the same saved 1,568-slot mask. The server audit records confirm final-mask equality for every pair.
The paired difference is +0.97 ± 1.09 percentage points, where ± denotes sample SD across seeds; four of five pairs are positive. Differences are calculated from correct counts before rounding displayed accuracies. All evaluations cover the same 1,319 GSM8K test questions under a common evaluator.
This gives a positive average answer in the tested setting: delayed removal can improve the final adapter even when its deployment identities and preceding history are fixed. Unlike replaying the final mask from initialization, the branches diverge only after their shared history. Retaining the buffer also increases active-capacity exposure during the remaining steps, so this is a test of the retention intervention rather than a compute-matched comparison. The result does not establish a benefit for every seed or a universally preferred commitment time.