acceptodds
Under review as a conference paper at ICLR 2027

Train Wide, Commit Late: Delayed Capacity Commitment for Parameter-Efficient Fine-Tuning

Abstract

Parameter-efficient fine-tuning (PEFT) typically makes the deployment budget binding before optimization ends. We challenge this coupling and introduce Delayed Capacity Commitment (DCC), a design principle that separates training-time capacity from final deployment commitment. DCC retains a small surplus of candidate capacity after adaptive compression and resolves it only at the end of training, without changing the final deployment budget. We instantiate DCC with Adaptive Competition Policy (ACP), which uses global rank-one competition and training-state-driven compression to govern capacity evolution. We call the paired effect of delayed versus immediate commitment under matched initial and deployment capacities the capacity dividend. On GSM8K, this dividend is consistently positive across five Qwen3-0.6B seeds, reaching +1.60 and +2.50 percentage points at the two tested buffer ratios; Llama-3.1-8B shows +1.95 points at the smaller buffer. Broader evaluations reveal substantial task-, model-, and budget-dependence. These results identify when capacity becomes binding as a distinct design dimension in PEFT.

Primary area:
optimization
Keywords:
Parameter-Efficient Fine-Tuning, Delayed Capacity Commitment, Low-Rank Adaptation, Capacity Dividend

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_X6vGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates whether low-rank parameter-efficient fine-tuning benefits from postponing the point at which the final deployment capacity becomes binding. It introduces Delayed Capacity Commitment, and the remaining surplus is removed only at the end of training to satisfy the deployment budget.

The proposed implementation, Adaptive Competition Policy, represents DoRA-style directional updates as globally competing rank-one slots. It combines importance scores derived from parameter magnitudes and optimizer statistics with magnitude and trend terms. A controller uses competition-state statistics and development-set feedback to govern compression. An important implementation detail is that both the surviving set and its selection ranking freeze when the intermediate floor is reached; final truncation uses that saved ranking.

The primary contribution is therefore an empirical investigation of a particular buffered capacity trajectory, together with a framework for describing and comparing such protocols.

Critical strengths
  • Well-motivated research question. The paper identifies an interesting and practically relevant question in PEFT.

  • Consistent improvements in the primary setting. The proposed approach achieves positive capacity dividends across all five paired seeds on Qwen3-0.6B GSM8K at both tested buffer ratios.

Critical weaknesses
  1. Limited novelty compared with existing adaptive-rank methods.

The idea of delaying deployment-budget commitment is related to existing adaptive-rank methods, such as AdaLoRA and SoRA. Although DCC introduces an intermediate compression floor, the difference from existing compression schedules is not sufficiently discussed. Moreover, the paper lacks a fair comparison with AdaLoRA under the same experimental settings, making it difficult to evaluate the actual improvement over existing methods.

  1. Insufficient ablation studies on ACP.

ACP contains several components, including importance scoring, global rank-one competition, and an adaptive compression controller. However, the paper does not provide sufficient ablation studies to analyze the effectiveness of each component. In addition, Tab. 4 shows that the simpler fixed rank-24 training with terminal selection can achieve comparable performance to buffered ACP. Therefore, it is unclear whether all these components are necessary.

  1. Limited generalization across different tasks and settings.

The performance improvements are mainly observed on GSM8K. As shown in Tab. 5, the improvements on BoolQ, PiQA, MAWPS, and AQuA-RAT are marginal, negative, or inconclusive. Moreover, a larger buffer ratio of 0.3 does not bring an average improvement on Llama-3.1-8B. These results indicate that the effectiveness of the proposed method is sensitive to different tasks and hyperparameter settings. More analysis is needed to explain under what conditions the proposed method can achieve consistent improvements.

Questions for authors
  1. What does the retained-buffer trajectory add beyond an established compression schedule?

  2. Does delaying buffer removal help when the preceding history and final mask are identical?

3 comments

Thank you for the two questions. We clarify the distinction between allocation and commitment below, and report the new experiment that directly holds the preceding history and final mask fixed.

Q1. What does the retained-buffer trajectory add beyond an established compression schedule?

ACP draws on DoRA's magnitude/direction parameterization and importance-based adaptive rank allocation. Its state-driven controller supplies a concrete implementation in which to investigate DCC. The research question is whether optimization should be restricted to the deployment budget as soon as adaptive allocation ends.

AdaLoRA's standard schedule reduces capacity to the target budget and then trains with the resulting allocation fixed (§3.3). Buffered ACP instead stops adaptive compression above that budget. For Qwen3-0.6B, the trajectory is 6,272 initial slots, 1,809 slots at the intermediate floor, and 1,568 deployment slots. The 241 surplus slots remain active and their weights continue updating until commitment.

An important implementation detail is that both the surviving set and the ranking used for final selection freeze at the intermediate floor. The deployment subset is therefore already identifiable then. DCC tests whether retaining the additional active capacity during subsequent optimization can still affect the final compact adapter. This does not require continued re-ranking after the floor.

A prescribed schedule can express the same trajectory. The contribution is the explicit separation and empirical comparison of allocation freeze and deployment-budget enforcement, rather than a claim that existing scheduling machinery cannot implement it. ACP is one realization of this comparison; the commitment question does not require its particular controller to be optimal.

The original within-ACP Qwen GSM8K comparisons give mean gains of +1.60 and +2.50 percentage points at the two buffer ratios, positive in all five paired seeds at each ratio. Those full-trajectory comparisons allow the final masks to differ. Your second question calls for a more specific control, which we have now completed.

Q2. Does delaying buffer removal help when the preceding history and final mask are identical?

In the new five-seed Qwen3-0.6B/GSM8K experiment, each run branches at its intermediate floor. One continuation removes the surplus immediately; the other retains it until training ends. They share the preceding training history and deploy the same saved 1,568-slot mask. The server audit records confirm final-mask equality for every pair.

SeedBranch stepDelayed removalEarly removalDelayed − early
4248038.89%36.85%+2.05 pp
12345039.04%39.65%−0.61 pp
340748038.06%37.30%+0.76 pp
9947040.26%38.29%+1.97 pp
77747037.91%37.23%+0.68 pp
Mean—38.83%37.86%+0.97 pp

The paired difference is +0.97 ± 1.09 percentage points, where ± denotes sample SD across seeds; four of five pairs are positive. Differences are calculated from correct counts before rounding displayed accuracies. All evaluations cover the same 1,319 GSM8K test questions under a common evaluator.

This gives a positive average answer in the tested setting: delayed removal can improve the final adapter even when its deployment identities and preceding history are fixed. Unlike replaying the final mask from initialization, the branches diverge only after their shared history. Retaining the buffer also increases active-capacity exposure during the remaining steps, so this is a test of the retention intervention rather than a compute-matched comparison. The result does not establish a benefit for every seed or a universally preferred commitment time.

Dear Reviewer,

As we approach the final hours of the author-reviewer discussion period, I would greatly appreciate it if you could take a moment to review and respond to the authors’ rebuttal.

Please briefly indicate which concerns have been addressed and which remain, and consider updating your scores if needed.

Thank you again for your time and valuable contributions!

Best regards, AC

Reviewer_X6vGOpenReview ↗

Response to the rebuttal

Thank you for the detailed rebuttal and additional experiments. The new controlled experiment helps clarify the effect of delayed commitment under identical training histories and final masks. However, I still have concerns regarding the limited novelty over existing adaptive-rank methods, insufficient validation of ACP's individual components, and inconsistent generalization across tasks and models. I appreciate the authors' efforts but will maintain my original rating.

Reviewer_zcpQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that efficient low-rank parameter fine-tuning (PEFT) should also determine when the deployment budget becomes a hard constraint during training. The paper proposes Delayed Capacity Commitment (DCC): adaptive compression stops when the deployment budget is exceeded, and excess capacity is only removed in the final step. The authors implement it as ACP, employing DoRA rank-one slots, a global competition mechanism, and a state-based controller. With the same initial and deployment capacities, delayed commitment outperforms immediate commitment on Qwen3-0.6B/GSM8K, but results are inconsistent in other settings.

Critical strengths

S1. The experimental design for endpoint matching is clean: delayed commitment shares the same initial and deployment capacities as immediate commitment.

S2. The method's internal gain on Qwen3-0.6B/GSM8K is robust: results are positive for all five random seeds under both major buffer ratios.

S3. Diagnostic control experiments are valuable, including active slot time matching, fixed mask replay.

Critical weaknesses

W1. The "commitment time" is essentially the time point at which the pre-defined schedule reaches the target budget and the structure is fixed in AdaLoRA-like methods; the pruning end time is itself a well-studied hyperparameter. The most distinctive claim in this paper is that committing in the last step without retraining yields better results, but this only works in the situation defined in the paper itself.

W2. Compared with fixed-rank DoRA, onlt have small and inconsistent improvement. On Qwen, the improvement ranges from 0.9 to 1.8 percentage points (depending on the buffer ratio), but different evaluation versions and factor scaling were mixed in the comparison; on Llama and with a deployment rank of 4, buffered ACP is lower than DoRA on every random seed.

W3. The paper's own control experiments show that the online mechanism of ACP is not necessary: ​​training with fixed rank 24 and performing a selection only at the end of training achieves comparable results to buffered ACP with almost the same active slot time.

W4. This effect is primarily reproduced on GSM8K. On BoolQ, PiQA, and MAWPS, the effect is close to zero or even negative. The result is not generalized.

Questions for authors

Refer to weakness.

2 comments

Thank you for the questions. Our central claim concerns when the deployment budget becomes binding during optimization. ACP provides a concrete implementation for studying this choice, rather than an assertion that its particular controller is the best way to realize DCC. All ± values below denote seed sample SD; differences use paired SD.

W1. Commitment versus the end of adaptive allocation. In AdaLoRA's standard schedule, the budget reaches its target and the resulting allocation is fixed for the remaining training (§3.3). DCC separates the end of adaptive allocation from enforcement of the deployment budget. For Qwen, the deployment budget is 1,568 global slots, corresponding to nominal average rank 8. Buffered ACP stops adaptive compression at 1,809 slots, continues training their weights, and removes the 241 surplus slots at commitment.

Both the surviving set and selection ranking freeze at the intermediate floor. Consequently, the final subset is already identifiable, but optimization still uses capacity above the deployment budget. The distinction is between fixing the allocation and restricting all subsequent optimization to the deployment-sized subset. A prescribed schedule can implement this separation; DCC makes it explicit and tests its consequence under matched initial and final capacities.

We added a five-seed GSM8K control that branches at the intermediate floor with identical preceding history and final masks. Retaining the surplus until the end gives +0.97 ± 1.09 percentage points over immediate removal, positive in four of five seeds. Final-mask equality is supported by server audits. This tests the post-branch retention intervention, although remaining capacity exposure is not matched.

Final-step commitment is a specified protocol, not an established optimum. The paper's three-seed timing sweep gives +2.91 ± 0.37 points for 100% versus 20% commitment, with non-monotonic intermediate means. Identifying a reliable rule for choosing commitment time remains open.

W2. Comparison with DoRA. We added Qwen3-0.6B experiments on MATH Level 1–2 with three seeds, using common training data/order and evaluation settings across the tested methods:

MethodMean accuracy
ACP, ρ=0.15440.10%
DoRA34.84%
AdaLoRA36.09%

ACP exceeds DoRA by +5.26 ± 2.26 points and AdaLoRA by +4.01 ± 1.15, with positive differences in all three seeds for each comparison. This supplies cross-method evidence beyond the small and evaluator-confounded historical GSM8K differences.

We agree that performance depends on the deployment budget and model. In the historical Qwen rank-4 pilot, buffered ACP improves over strict ACP by +1.74 ± 0.77 points but remains 4.35 points below DoRA. On Llama GSM8K at rank 8, buffered ACP reaches 59.69% against DoRA's 61.33%. The new MATH result supports the tested configuration, not a general ranking of methods or superiority over fully tuned baselines.

W3. Fixed rank 24 and the role of ACP. Fixed rank 24 with terminal selection reaches the buffered accuracy range: 40.89% versus 40.31%, at similar mean active-slot counts. We regard this as evidence that simpler training-above-deployment protocols deserve consideration. It does not independently isolate commitment timing, because the initial widths and training trajectories differ.

Our primary DCC evidence comes from within-ACP interventions: the paper's two buffer settings improve over strict ACP in all five paired Qwen GSM8K seeds, and the new fork directly tests surplus retention after a shared history with fixed deployment identities. These comparisons concern the commitment protocol, rather than the necessity of every scoring or control component. ACP automates intermediate compression using training-state signals; finding simpler and more effective realizations of DCC remains a research question.

W4. Generalization and the timing of late commitment. We added Qwen3-0.6B commitment-time sweeps on MATH Level 1–2 and AQuA-RAT rationale (three seeds, 630 steps, 2,048-token evaluation cap):

TaskCommit 40%60%80%100%
MATH39.35%40.35%42.11%40.10%
AQuA-RAT36.88%37.40%36.88%38.19%

MATH's highest tested mean occurs at 80%, and AQuA-RAT's at 100%. In all three seeds of each of these settings, the 1,809-slot allocation had already frozen before step 504 (80%). Both therefore retain surplus capacity beyond allocation freeze: late commitment need not mean committing only at training's end.

Our main experiments use 100% commitment as a simple common protocol, not an established optimum. These sweeps show different mean responses across tasks; final masks can also differ across timing arms. Choosing the best post-freeze commitment time for a new task remains an open question.

Dear Reviewer,

As we approach the final hours of the author-reviewer discussion period, I would greatly appreciate it if you could take a moment to review and respond to the authors’ rebuttal.

Please briefly indicate which concerns have been addressed and which remain, and consider updating your scores if needed.

Thank you again for your time and valuable contributions!

Best regards, AC

Reviewer_tVyzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies when a low-rank adapter should reach its final deployment capacity during fine-tuning. Delayed Capacity Commitment retains a small capacity buffer until training ends, then removes the surplus. Adaptive Competition Policy implements this with global slot scoring and a controller for progressive compression. Matched comparisons show positive accuracy differences on GSM8K for Qwen3-0.6B and one Llama-3.1-8B setting. Additional controls show that the effect depends on task, model, buffer size and deployment rank, without identifying one mechanism for the gain.

Critical strengths
  1. The paper turns commitment timing into an explicit, testable design choice. Sections 3.3 and 4.2 compare immediate and delayed commitment within the same framework, with matched initial and final slot counts. Table 1 shows useful GSM8K improvements, and Section 4.3 directly varies the commitment point. This is stronger evidence than simply comparing an elaborate adaptive method with a fixed-rank baseline.

  2. The diagnosis is careful and candid. Section 4.4 examines training exposure, final masks, compression rules and training width. Section 4.5 reports mixed transfer results, and Appendix E.5 separates evaluation snapshots. The authors acknowledge that these controls do not establish a unique mechanism or a universal benefit. This makes the limited empirical claim easier to trust.

Critical weaknesses
  1. I am not yet convinced that adaptive partial compression adds enough beyond training wider and compressing at the end. Table 4 shows that fixed rank-24 training can reach the buffered ACP accuracy range, and Table 3 shows that terminal score selection itself has a substantial effect. Section 2 already acknowledges the established train-wide-then-compress idea. I find the within-ACP timing comparison useful, but the reported controls leave me unclear about the additional insight or practical benefit of the intermediate compression policy. I view this as a limit on the significance of the contribution and do not take it as evidence that the reported dividend is incorrect.

  2. I am not sure how to interpret the matched slot count as a practical deployment constraint. Equation 4 correctly explains that different module dimensions allow equal slot counts to correspond to different compact parameter counts. Appendix A.2 also notes that masked tensors can retain their original storage size. I could not find the realized compact sizes of the immediate and delayed adapters for the main comparison. These values matter to my interpretation of the gain under a common deployment constraint. I see this as a fixable reporting issue that the authors can address using the existing final masks.

Questions for authors
  1. I would like to know what Tables 3 and 4 support about adaptive partial compression beyond the existing idea of training wider and selecting a compact adapter at the end. I would like the authors to distinguish the demonstrated result from the proposed design principle.

  2. I would like the authors to calculate the realized compact parameter counts for both arms of the main matched comparison using the stored final masks, and clarify which deployment resource is actually matched.

2 comments

Thank you for separating the observed capacity dividend from the practical value of ACP's intermediate compression. Our contribution concerns when the deployment budget constrains optimization; ACP is the implementation used to investigate that choice.

Why this ACP design? ACP combines DoRA's magnitude/direction parameterization with importance-based allocation across rank-one slots, drawing on adaptive-rank methods. Scores combine base importance, magnitude and trend terms with EMA smoothing. Its controller uses winner-set stability, score separation, and development feedback to govern compression. This permits a comparison between compressing to the deployment budget and stopping above it while holding the initial and final slot counts fixed. At the intermediate floor, both the surviving set and selection ranking freeze; weights continue training. Final removal uses that saved ranking. Thus, the retained buffer supplies additional active training capacity even after the final subset is identifiable.

The practical value of the intermediate adaptive policy remains open. Table 4 shows fixed rank 24 with terminal selection at 40.89%, versus buffered ACP at 40.31%, with similar mean active slots (4,704 versus 4,637). This is consistent with the broader value of training above deployment capacity, but does not independently isolate commitment timing. Table 3 shows a separate effect: score selection exceeds matched-rank SVD by 3.16 ± 0.22 points from the same rank-24 source weights. Neither establishes an advantage for the intermediate adaptive policy.

What new evidence tests commitment itself? The original within-ACP comparison gives +1.60 and +2.50 points on Qwen GSM8K, positive in all five paired seeds at each buffer ratio. We now additionally branch five runs at the intermediate floor, keeping the preceding history and final 1,568-slot mask identical. One branch removes the surplus immediately; the other retains it until training ends. Delayed-minus-early differences are +2.05, −0.61, +0.76, +1.97, and +0.68 points: +0.97 ± 1.09, positive in four of five seeds. Final-mask equality is supported by saved server audits. This directly tests continued surplus participation after the subset is known, though remaining capacity exposure is not matched. All ± values denote seed sample SD; differences use paired SD.

Deployment-budget fairness. All main Qwen arms start with 6,272 slots and deploy exactly 1,568. We counted compact parameters from the exported masks and tensor shapes of all 15 evaluated adapters, including retained input/output vectors and DoRA magnitudes:

Pcompact=∑ℓrℓ(din,ℓ+dout,ℓ)+∑ℓdout,ℓ.P_{\mathrm{compact}}=\sum_\ell r_\ell(d_{\mathrm{in},\ell}+d_{\mathrm{out},\ell})+\sum_\ell d_{\mathrm{out},\ell}.

SeedImmediate, ρ=0Delayed, ρ=0.154Delayed, ρ=0.3
425,415,9365,415,9365,415,936
1235,417,9845,414,9125,422,080
34075,411,8405,424,1285,417,984
995,410,8165,408,7685,409,792
7775,415,9365,424,1285,421,056

The largest absolute paired difference is 12,288 parameters (0.23%). Seed 42 has exactly equal counts, with gains of 2.12 and 3.49 points for the delayed arms. Across ten pairs, delayed adapters have equal counts in two, fewer parameters in three, and more in five; all ten accuracy differences are positive. The exact matched resource is global slot count, with small measured deviations in compact scalar counts. These counts do not establish equal checkpoint-file sizes or inference speed: the masked implementation can retain full-size tensors.

Dear Reviewer,

As we approach the final hours of the author-reviewer discussion period, I would greatly appreciate it if you could take a moment to review and respond to the authors’ rebuttal.

Please briefly indicate which concerns have been addressed and which remain, and consider updating your scores if needed.

Thank you again for your time and valuable contributions!

Best regards,
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.