acceptodds
Under review as a conference paper at ICLR 2027

Understanding Update Selection in Low-Rank Fine-Tuning

Abstract

Low-rank adaptation (LoRA) is commonly evaluated through final benchmark scores, which do not isolate the effects of individual training updates. We study whether additional evidence improves one-step update selection and what explains the resulting gains. Holding the model and optimizer history fixed, we compare candidate updates and measure their one-step cross-entropy reduction on held-out examples. Locally improving updates can harm held-out data, and proposals with matched effective-weight displacement norms can have different held-out gains. A frozen selection rule using additional task-balanced examples achieves mean gain 0.002071 on previously unused examples with Llama-3.2-1B, compared with -0.001090 for full updates and -0.000011 for current-loss selection. A Qwen3-4B-Base replication also shows higher mean gain with additional evidence. Reassigning choices within trajectories reduces gain while preserving candidate frequencies and acceptance rates, showing that overall preferences and fewer updates do not fully explain the benefit. Preserving acceptance locations retains much of the gain. On Qwen, simpler evidence-based policies and controls that preserve training step also retain much of the benefit. These results provide a controlled account of the gains from one-step update selection in low-rank fine-tuning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Low-Rank Adaptation, Parameter-Efficient Fine-Tuning, Update Selection, Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_mQkiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies one-step update selection in LoRA fine-tuning through fixed-state comparisons. It restores model parameters, current gradients, and optimizer history before evaluating alternative updates, including skipping, scaling the source proposal, changing factor learning-rate ratios, and matching effective-weight displacement norms. The experiments separate candidate construction, selection evidence, and held-out evaluation. Additional task-balanced examples improve mean one-step held-out cross-entropy gain on Llama and Qwen. Retrospective reassignment controls further examine whether these gains arise from overall candidate preferences, acceptance locations, or more detailed state–choice matching.

Critical strengths
  • The paper offers a fresh perspective by studying one-step update selection in LoRA fine-tuning.

  • The experimental design controls important confounders. Restoring optimizer moments and step counts, rather than only model parameters, makes the candidate comparisons substantially cleaner. Separating gradient-generation data, selection evidence, and evaluation examples is also appropriate.

  • The reassignment controls are informative. Preserving candidate frequencies and acceptance rates while changing their assignment to states provides evidence beyond a simple comparison of average acceptance rates. The acceptance-preserving and same-step controls help qualify the interpretation.

Critical weaknesses
  • The paper appears to be primarily exploratory. After reading it in full, I am still unclear about its central takeaway, as it does not seem to reach a clear conclusion.

  • The strongest result is that additional target-relevant examples improve selection over current-batch loss. However, on Qwen, acceptance-only and validation line search already achieve gains of 0.002099 and 0.002097, versus 0.002558 for full selection; the paired advantages of full selection have intervals containing zero. Moreover, same-step cross-seed reassignment retains a gain of 0.002546, close to the original 0.002558. The paper leaves unclear how much decision complexity or state-specific information is actually necessary.

  • The authors sample states from existing training trajectories, compare alternative one-step updates, and then restore the original state. If the selector were applied repeatedly during training, subsequent parameters, optimizer states, and gradients would all change. The paper does not establish whether sustained use of this selection strategy would produce a better final model.

Questions for authors
  • The experiments focus primarily on small-scale multitask binary classification fine-tuning. Why has the evaluation not been extended to long-form text generation, reasoning training, or knowledge distillation?

  • How stable are the Llama confirmation gains across independently drawn selection panels and additional training trajectories? Can uncertainty be reported over these sources of variation?

2 comments
AuthorsOpenReview ↗

Response to Reviewer mQki

  1. Question: Why was the evaluation not extended to generation, reasoning training, or knowledge distillation?

    Response: The classification setting fixes model and optimizer state, supports exact token-level CE comparisons, and allows disjoint selection and evaluation examples without decoding variability. This directly tests the paper's controlled local question; it is not intended to exhaust all downstream settings. Long-form generation, reasoning training, and distillation use different objectives and evaluation protocols and are natural extensions. A paper need not cover every dataset, seed, parameterization, and downstream task to establish a controlled result on its stated benchmark; our claim is limited to the evaluated scope and does not make a universal generation claim.

  2. Question: How stable are the gains across independent panels and additional trajectories?

    Response: The reported uncertainty intervals have different scopes. For Llama, the original mean-gain 95% bootstrap interval is [0.000449, 0.003731], and the paired interval for Original minus within-trajectory reassignment is [0.001233, 0.002917]. These intervals are conditional on the observed tasks, source states, and selection panel; they do not quantify new trajectories or newly sampled panels. For Qwen, resampling includes three seed blocks, five selection-panel draws, and evaluation examples. The paired gaps are 0.630e-3 [0.357e-3, 0.910e-3] for all-choice reassignment, 0.073e-3 [0.011e-3, 0.158e-3] for accepted-only reassignment, and 0.013e-3 [-0.016e-3, 0.045e-3] for same-step cross-seed reassignment. The last interval contains zero, so we do not claim that finer state matching is established. The corrected full-trajectory test adds a further caution: fixed panel gives Q=1033 versus 1031 for Full, while current-batch gives 1029 under the same protocol.

  3. Question: Do simpler policies already capture the gain?

    Response: Yes, the Qwen one-step results separate the usefulness of additional evidence from the necessity of the richest candidate menu. Acceptance-only and validation line search obtain mean gains of 0.002099 and 0.002097, compared with 0.002558 for full selection, and the paired advantages of full selection have intervals containing zero. The evidence therefore supports the narrower claim that evidence beyond the current batch can be useful.

Thank the authors for their response. I decide to maintain the score because I think my main concerns have not been adequately addressed.

Reviewer_aHNkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to choose a single LoRA update using evidence beyond the batch that produced its gradient. The authors restore the adapter parameters, current gradient, and AdamW history before comparing nine declared candidates, including skipping the update, reducing its size, changing the relative learning rates of the two factors, and matching effective-weight displacement norms. Selection and outcome evaluation use separate examples. In the Llama-3.2-1B confirmation experiment, a frozen selector using 192 task-balanced examples achieves a mean held-out cross-entropy reduction of 0.002071 across 120 source states, compared with -0.001090 for full updates and -0.000011 for current-loss selection. A Qwen3-4B-Base replication also favors additional selection evidence. Retrospective reassignment controls show that candidate frequencies and acceptance rates alone do not explain the gains, although preserving acceptance locations retains much of the benefit. On Qwen, simple evidence-based policies and a control preserving training step also recover much of the gain.

Critical strengths
  • The fixed-state protocol makes the local comparisons credible. Every candidate starts from the same LoRA parameters, clipped gradient, AdamW moments, and step count, with zero reported replay errors. Restoring optimizer history is particularly important because it prevents differences inherited from earlier training from being mistaken for effects of the current update.

  • The additional evidence produces a positive mean Llama gain of 0.002071 and outperforms full, half-step, and current-loss selection in all ten source groups (Table 4). The Qwen replication also shows improvement even though full updates already have positive average gain there. Together, these results support the usefulness of selection evidence across different source configurations and two backbones.

Critical weaknesses
  • One detail I would like to understand better is how often the acceptance-location control actually changes a decision. With four sampled states per trajectory, fewer than two accepted states or repeated candidate identities can leave choices unchanged. Reporting how many trajectories contain distinct accepted choices, together with results for that subset, would help assess how strongly the retained gain supports the interpretation about acceptance locations.

  • The evidence comparison supports the combined selection procedure, although the contribution of each ingredient is somewhat harder to assess. Current-loss selection uses the 64 examples that generated the gradient, while the balanced selector uses 192 additional examples matching the target task mixture. Comparisons at matched budgets and controlled task mixtures could help distinguish the effects of sample size, fresh examples, and task composition.

Questions for authors

Please see the weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer aHNk

  1. Question: How often does the acceptance-location control actually change a decision?

    Response: The control reassigns candidate identities across the four sampled states within each trajectory while keeping rejected states as no update. An assignment can change only when a trajectory has at least two accepted states with different choices; trajectories with fewer than two accepted states are unchanged by construction. The control gain is computed by enumerating indexed permutations rather than one random shuffle. The aggregate original-to-accepted-location values are 2.071 to 1.642 for Llama and 2.558 to 2.485 for Qwen. We use this aggregate control to test the interpretation; the subset count is descriptive and is not treated as a separate causal estimate.

  2. Question: Can sample count, task mixture, and fresh evidence be separated at matched budgets?

    Response: The original comparisons test related evidence designs rather than claiming a complete factorial decomposition. At a fixed budget of 48 examples, task-balanced evidence outperformed a label-balanced selector matching current-batch task proportions in 8 of 10 source groups; because the examples also differ, this is evidence about the tested designs rather than an isolated task-composition effect. Increasing the balanced budget from 192 to 768 did not improve every group, which argues against “more examples is always better.” The primary conclusion is narrower: under the stated protocol, additional target-relevant selection evidence improves mean one-step utility. A same-example, matched-budget factorial study would answer a finer attribution question but is not required for that conclusion.

Reviewer_Dt6pOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies one-step update selection in low-rank fine-tuning (LoRA). Rather than evaluating only the final performance of fine-tuned models, the authors analyze the effects of individual optimization updates on held-out data. They construct multiple candidate updates from the same model parameters, gradients, and optimizer history, including scaled updates, different relative learning rates for LoRA factors, and no update. The candidates are then compared using cross-entropy loss on additional examples. A frozen selection rule using additional task-balanced examples achieves meaningful performance gain on both Llama and Qwen models.

Critical strengths
  • The paper is written in a way that is easy to follow.
  • The idea of choosing an immediate LoRA’s update is reasonable and intuitive.
  • The performance gain of the proposed framework is promising.
Critical weaknesses
  • The connection between LoRA's immediate update choice and the performance is unclear in theoretical perspective.
  • The computation cost-accuracy trade-off of the proposed framework is unclear.
  • Limited novelty and insufficient insight into the underlying mechanism.
  • Limited experimental scope.
Questions for authors
  • Can the authors provide a deeper analysis of the observations from a theoretical perspective? Why is the selection rule better?
  • What is the performance of the selection rule when combined with other LoRA variants such as AdaLoRA [1], LoRA-Pro[2],...?
  • Can the authors provide the complexity analysis of the proposed framework?
  • What is the computation cost-accuracy trade-off of the proposed framework?
  • Can the authors provide experimental results on more challenging settings, such as instruction tuning and language generation?

[1] AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. [2] LoRA-Pro: Are Low-Rank Adapters Properly Optimized?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Dt6p

  1. Question: Can the authors provide a deeper theoretical analysis of why the selection rule is better?

    Response: Let gTg_T and gQg_Q denote the current-batch and target-distribution gradients. For a candidate update δ\delta,

    LQ(θ+δ)−LQ(θ)≈gQ⊤δ+12δ⊤HQδ.L_Q(\theta+\delta)-L_Q(\theta) \approx g_Q^\top\delta + \tfrac{1}{2}\delta^\top H_Q\delta.

    The current batch can be a noisy or task-mismatched estimator of gQg_Q. A separate task-balanced selection set supplies another estimate of the target directional derivative and can therefore change candidate rankings even when all candidates start from the same state. This is a local transfer explanation, not a guarantee; a selected candidate can still have negative held-out gain, which is why harm and acceptance are reported. For LoRA, with W=W0+γBAW=W_0+\gamma BA, the effective update is ΔW=γ[(B+ΔB)(A+ΔA)−BA],\Delta W=\gamma\left[(B+\Delta B)(A+\Delta A)-BA\right], so equal factor displacement need not imply equal direction or target-loss change. These arguments explain the fixed-state observations but do not constitute a global convergence theorem.

  2. Question: What happens when the method is combined with AdaLoRA or LoRA-Pro?

    Response: AdaLoRA and LoRA-Pro are complementary method-level variants, not necessary controls for the fixed-state question. The protocol is parameterization-agnostic at its interface: it restores method-specific parameters and optimizer state, constructs a declared candidate set, and scores the resulting candidates. Their rank allocation, factor geometry, and optimizer state can change quantitative outcomes, so a combination result is a separate experiment rather than a prerequisite for the present controlled claim.

    More broadly, a comparison to LoRA-Pro or any other LoRA variant should be interpreted only after aligning SFT masking, prompt/template formatting, batch and accumulation settings, learning-rate search, checkpoint selection, and evaluation. We verified that the reproduced Llama-2 training settings are broadly aligned (including response-only masking, Alpaca-style prompting, sequence length 1024, micro-batch 1, accumulation 32, one epoch, cosine decay, and 0.03 warmup), so we do not attribute the published gap specifically to mask or prompt. However, the LoRA-Pro table still mixes its own Full FT evaluation path with inherited LoRA/LoRA-GA numbers, and the public evaluation scripts use different generation settings. A large gap under that mixed provenance cannot be interpreted as a pure adapter-update effect; a single shared learning rate may also miss one method's peak. Our fixed-state controls are designed to make the update-selection component comparable before making a broader peak-performance claim.

  3. Question: What is the complexity of the framework?

    Response: At one decision point, candidate scoring requires approximately KK post-update evaluations on MM selection examples, plus candidate construction and state restoration. Over TT steps with cadence cc, the dominant evaluation term is

    O(TcKM).O\left(\frac{T}{c}KM\right).

    The base LoRA optimizer update is unchanged; the additional asymptotic cost is in candidate evaluation and restoration, and candidates can be evaluated sequentially. The complete-trajectory cost--accuracy table reported in the QTeU computation--performance response gives the corresponding measured overhead.

  4. Question: What is the computation cost--accuracy trade-off?

    Response: The complete-trajectory cost--accuracy table reported in the QTeU computation--performance response gives the direct trade-off: Fixed-panel selection takes 2.42x the Fixed Full runtime and increases Q from 1031 to 1033, while current-batch selection takes 2.43x and decreases Q to 1029. Candidate-menu size, cadence, and evidence budget determine this cost--utility trade-off.

  5. Question: Can the method be evaluated on instruction tuning or language generation?

    Response: Classification is used because it permits exact fixed-state replay, disjoint selection/evaluation examples, and precise one-step CE accounting without decoding variability. Instruction tuning and generation use broader objectives and introduce choices such as sequence length, decoding, and sampled prefixes. The protocol can be adapted to token-level sequence loss and generation-specific evaluation, but the current experiment should not be presented as already covering generation.

Thank the authors for their response. I decided to maintain the score, as the main concerns about the theoretical analysis and limited experimental scope are not fully addressed.

Reviewer_BjBJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper considers the problem of update selection over a one-step update window during low-rank adaptation (LoRA) fine-tuning process. The main contribution of this work are the experimental results relating to selecting most beneficial updates during LoRA training and evaluating single step cross-entropy (CE) testing loss. The authors consider two pretrained models in their work: Llama-3.2-1B and Qwen3-4B-Base, and fine tune these models on six binary classification tasks for a total of 192 steps using the AdamW optimizer and batch size of 64. The authors save the state of the LoRA fine-tuning process at steps 10, 60, 110, 160 (where the state consists of model parameters, training batch gradients, and AdamW moments) and for every such state, they evaluate nine potential candidate updates for a single update step, based on different criteria (no update, scaling the step size of the update, and different learning rate ratios of the B and A matrices of the LoRA update as well as their norm-matched versions). Each update is then evaluated on a task-balanced validation set of 192 sequences (four tasks, 48 sequences each) and the update that achieves the lowest cross entropy (CE) loss is selected as the best update. Such chosen update is then compared on a task-balanced test set of 256 sequences (four tasks, 64 sequences each) against various proposed baseline policies, such as no update, regular full update, updating with the step size scaled by 0.5, and selecting the best update based on the training batch. The results demonstrate that the proposed update policy achieves best test-set performance across different evaluated configurations.

Critical strengths

None. While the experiments are clearly described and reproducible, I don't believe the demonstrated contribution of the paper meets acceptance bar for ICLR.

Critical weaknesses

While the paper has a lot of weaknesses, the two main ones would be:

  • Not applying the proposed policy to the entire LoRA update process. The authors test their proposed policy only on 4 frozen states at LoRA update steps 10, 60, 110, 160, discarding the policy-based update after each evaluation on the testing set. Furthermore, the testing is also only done once per each state. As a result, while the authors demonstrate that choosing an update using validation data reduces loss on a held-out testing set for a given LoRA step, this does not at all translate to the proposed policy achieving best (lowest CE loss/highest task accuracy) final fine-tuned model if it was applied over the entire course of the training. In fact, applying this policy repeatedly could lead to a set of potential issues, such as training instability, single update rule becoming dominant (and therefore a reasoning for only using it instead of comparing nine candidates at every step), introduction of bias, etc, none of which are considered or tested by the authors. Also, I acknowledge that, as the authors claim, running the entire fine-tuning process does not "isolate the effect of the current update," and while this is a valid justification for evaluating the proposed policy on single-step updates, the goal of LoRA fine-tuning is to arrive at a good fine-tuned model, so any policy that proposes selecting better updates should be tested whether it results in creation of a better final model.
  • The contribution of the paper is very limited and doesn't provide a novel or even useful enough insight. The authors state three main contributions, with the first being "We compare alternative LoRA updates from the same training state and show that current-batch improvement and effective-weight displacement do not determine held-out utility," which means that an update based on a non-task-balanced training batch will perform worse on a task-balanced testing set than an update chosen conditioned on a task-balanced validation set. This is an expected property of any multi-task optimization process and does not constitute a novel insight into LoRA. Similarly, the second contribution claims "A frozen selection procedure using additional examples improves mean held-out loss reduction on unused Llama evaluation data, with the same pattern observed in a Qwen replication," which is almost guaranteed given that the validation set has a larger size and the same task distribution as the testing set, unlike the training batch, so one would naturally expect a selection procedure based on such set to have a lower testing loss. Similar argument is true for the remaining contribution and the demonstrated results.
Questions for authors

Several questions worth considering:

  • What would the effect of applying such a procedure be if applied over the entire fine-tuning trajectory? What are some possible issues that could arise when doing that?
  • If such procedure was applied during LoRA fine tuning, what would be the computational overhead associated with it? (The authors mentioned the time it takes to score all candidates, but I would also be interested in the total overhead time of the proposed policy)
  • Why the specific 9 procedures were chosen? Why not consider other types of modifications to the LoRA update? And why 9 specifically? Is there a reason that the specific step size scales were selected? Or why more ratios of the LoRA parameters were not considered?
  • Why were the steps 10, 60, 110, 160 chosen specifically? And in the directions, for these steps, which candidates are chosen? What is the distribution of the chosen candidate updates? If we ran it for say more steps (say one update evaluation every 5 steps on a set trajectory), would the chosen candidates be distributed uniformly? Or would certain updates be more common?
  • Why are some results missing? (Table 5a Llama)
2 comments
AuthorsOpenReview ↗

Response to Reviewer BjBJ

  1. Question: What happens when the procedure is applied over the entire fine-tuning trajectory, and what problems could arise?

    Response: We tested repeated application with the original nine candidates, Adam/RNG/state semantics, selection every five steps, and 819 steps. All runs completed with finite losses and without NaN/OOM; all nine candidates were selected at least once, and endpoints were frozen before S/D/Q scoring. The fixed-panel selector accepted 153/164 decisions and reached Q=1033/1221 versus 1031/1221 for Fixed Full, with S/D=209/256 and 229/256. The current-batch selector accepted 162/164 decisions but reached Q=1029/1221. The fixed-panel run had entropy 3.129 bits and maximum repeated-candidate length 3, so it did not collapse to one candidate; the current-batch run was more concentrated (entropy 2.259 bits, maximum repeat length 9). Repeated selection is therefore executable and changes the trajectory, but the observed Q gain is small, does not transfer to S, and is not reproduced by the current-batch control.

  2. Question: What is the total computational overhead?

    Response: The end-to-end cost--accuracy table in the QTeU computation--performance response gives the complete-trajectory overhead: the fixed-panel selector takes 2.42x the Fixed Full runtime and the current-batch selector takes 2.43x. With K candidates, M selection examples, T optimization steps, and selection cadence c, the dominant scoring term scales as O((T/c)KM), in addition to candidate construction and state restoration.

  3. Question: Why were these nine candidates, scales, and factor ratios chosen?

    Response: The menu is a small, predeclared decomposition of four questions: (i) whether to update at all (no update); (ii) whether the source step is too large (source fractions 1/16, 1/8, 1/4, 1/2); (iii) whether the relative learning rates of the two LoRA factors matter (raw ratios 1 and 16); and (iv) whether that factor-ratio effect remains after matching effective-weight displacement (the two norm-matched variants). The source fractions are geometrically spaced, while ratios 1 and 16 provide a clear equal-rate versus asymmetric contrast. Nine is a controlled candidate menu, not an exhaustive search over update rules, and the claims are limited to this declared decomposition.

  4. Question: Why were steps 10, 60, 110, and 160 chosen, and how are choices distributed under denser selection?

    Response: The four original states are evenly spaced over the 192-step source trajectory and cover early, middle, and late training without using evaluation outcomes to select states. In the corrected 819-step stress test with selection every five steps, the fixed-panel selector used all nine candidates: no update 11 times; source fractions 1/16, 1/8, 1/4, and 1/2 16, 27, 20, and 18 times; raw ratios 1 and 16 22 and 14 times; and matched ratios 1 and 16 19 and 17 times. Choices were non-uniform but did not collapse (entropy 3.129 bits, maximum repeat length 3). The current-batch control also used all nine candidates but was more concentrated (entropy 2.259 bits, maximum repeat length 9). Non-uniform frequencies are expected state-dependent behavior, rather than a requirement that the selector be uniform.

  5. Question: Why are some Llama results missing from Table 5(a)?

    Response: This is a presentation issue. The Llama mean gains in Table 5(a), in 10^-3 CE units, are: Original 2.071, Reassign all -0.002, Fix accepted states 1.642, and Best fixed 1.172. Same-step cross-seed is not applicable to Llama and is marked “—”; it is reported for Qwen only. We repeat these values here and distinguish “not applicable” from missing data in the table.

Thank you for the clarifications about my questions. While these clarify some of the design decisions, several significant issues remain. For instance, the significant computation overhead greatly increases the runtime, which would be a strong limiting factor in practical deployment. The limited candidate selection set remains problematic, especially when considering that, as per the response to question 1, all candidate updates were selected, which leads to further questions about whether other, non-evaluated update procedures or parameters can better improve LoRA performance without the proposed framework. As significant concerns remain about the paper's applicability and scope, I decided to maintain my score.

Reviewer_QTeUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies if held-out examples can inform the selection of LoRA modules updates at a fixed state on the training trajectory. The candidate updates are constructed by applying scaling and different learning rate ratios. Experiments on Llama and Qwen models show that selecting updates based on additional samples can improve cross-entropy gain on the held-out data over full updates.

Critical strengths

Controlled experimental setup: Comparing candidate updates from the same training state, with optimizer history held fixed seems to be a reasonable way to isolate the effect of update selection.

Critical weaknesses

Poor writing and unclear motivation. The paper is difficult to follow and does not show a clear motivation for the problem. The abstract does not tell a coherent story and explain the significance of the proposed study. The introduction introduces several concepts without sufficient exposition. For example, the first paragraph in the introduction refers to reducing an update, changing the relative updates of two factors, or skipping an update, without first explaining why these decisions matter. The term "controls" is also introduced without sufficient clarification of what it refers to in this context. Overall, the presentation makes it difficult to understand the problem setting and the intended contribution.

Limited novelty and practical significance. The paper considers an update selection problem at a prior training state. However, the practical relevance of this setting is not clear to me. In particular, the candidate updates are generated from models that have already undergone training or fine-tuning. At that point, the training process has already produced information that could be used to assess candidate quality, including downstream or validation performance. It is therefore unclear when the slection procedure would be needed in practice. Moreover, what advantage does it offer over simply evaluating the candidate fine-tuning configurations?

The paper also does not establish that this particular update selection problem is sufficiently important to be a standalone contribution. The experiments primarily demonstrate that additional target-relevant examples can help select among a predefined set of candidate updates, which is not by itself a particularly surprising observation.

Insufficient experimental evaluations. The experiments lacks comparisons with established update-selection or adaptive optimization baselines. The paper compares against full updates, smaller updates, and current-loss selection, but these are relatively limited reference policies. It would be important to compare against more principled alternatives. Additionally, the selection procedure requires evaluating multiple candidate updates on additional examples. This will incur addtional computational costs. The paper should discuss the trade-off between the performance improvement and additional runtime / computational costs.

Questions for authors

See above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer QTeU

  1. Question: The motivation and central problem are unclear.

    Response: The central question differs from final-checkpoint comparison: given the same LoRA parameters, current gradient, AdamW state, and candidate set, can task-relevant evidence identify a better next update than the current training batch? Fixing the source state identifies the effect of selection evidence. The training batch constructs candidates, a separate selection set chooses among them, and a disjoint evaluation set measures one-step utility. The contribution is update utility and evidence design, rather than a new LoRA parameterization.

  2. Question: When would the procedure be useful, and why not simply evaluate complete candidate fine-tuning configurations?

    Response: The procedure applies to multi-step fine-tuning when a small task-relevant evidence set is evaluated at selected update points, including fixed-cadence offline training. Training a complete trajectory for every candidate would require a separate trajectory for each candidate, whereas fixed-state selection evaluates alternatives from one restored state. In the corrected pure-CE full-trajectory test (nine candidates, selection every five steps, 819 steps), Fixed Full reached Q=1031/1221, the fixed-panel selector Q=1033/1221, and the current-batch selector Q=1029/1221. The panel selector changed the trajectory and increased Q by 2, while S changed from 213 to 209; current-batch did not reproduce the increase.

  3. Question: The novelty and practical significance appear limited.

    Response: The contribution decomposes three factors: selection evidence, candidate updates available at a fixed state, and assignment of choices to states. It is also a fairness contribution because source state, optimizer state, source batch, candidate menu, and evaluation rule are identical in the method-only comparison. End-to-end comparisons should give every method the same tuning budget, held-out checkpoint selection, and evaluation path, and compare peak performance.

    LoRA-Pro Table 2 illustrates this issue. The table reports 5.30/59.36/35.31 for Full FT, whereas the original LoRA-GA paper reports 5.56/54.20/19.87. The LoRA and LoRA-GA rows in LoRA-Pro match the LoRA-GA results entry by entry, including standard deviations. The table therefore places LoRA-Pro's Full FT results alongside LoRA and LoRA-GA results carried over from LoRA-GA, rather than rerunning every method through one shared evaluation code path. The public scripts also use different generation and sampling settings. This pattern propagates: GoRA adopts its CLIP baseline from LoRA-Pro, and CDKA uses non-starred GoRA results as inherited baselines. Later comparisons therefore need aligned source code, training protocol, tuning budget, checkpoint selection, and evaluation path.

    On Llama, frozen balanced selection gives mean gain 0.002071 versus -0.001090 for Full and -0.000011 for current-loss selection. On Qwen, the corresponding values are 0.002558, 0.001453, and 0.001996. Reassigning candidate identities while preserving frequencies and acceptance rates reduces the gain, while fixing accepted locations retains much of it. These controls identify the evidence, state, and assignment factors behind update utility.

  4. Question: More principled baselines and the computation--performance trade-off are needed.

    Response: The fixed-state protocol compares no update, Full, half-step, current-loss selection, acceptance-only, validation line search, candidate reassignment, and same-step cross-seed reassignment. These controls separate evidence, choice, and assignment effects and keep the method-only comparison on a common code path.

    The end-to-end cost--accuracy result is summarized below.

    PolicyQTotal runtimeRelative runtimeS / D
    Fixed Full1031/1221676.2 s1.00x213 / 228
    Fixed panel selector1033/12211,636.5 s2.42x209 / 229
    Current-batch selector1029/12211,642.9 s2.43x210 / 227

    The panel selector increases Q by 2 under this protocol, while current-batch decreases Q by 2. Candidate-menu size, cadence, and evidence budget determine the cost--utility trade-off.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall response

Why this work matters: fair attribution of LoRA improvements

We appreciate the reviewers' detailed questions. They point to a central methodological issue: a reported gap between two LoRA methods is attributable to their update rules only when the surrounding training and evaluation protocol is aligned. A fair comparison aligns the training objective and label mask, prompt/template format, data and packing, batch size or gradient accumulation, optimizer and schedule, rank and parameter budget, checkpoint selection, and evaluation implementation. It also gives every method the same tuning budget and compares peak performance, with checkpoint selection on held-out data.

LoRA-Pro Table 2 illustrates this methodological issue clearly. The table reports 5.30/59.36/35.31 for Full FT, whereas the Full FT results in the original LoRA-GA paper are 5.56/54.20/19.87. At the same time, the LoRA and LoRA-GA rows in LoRA-Pro match the corresponding LoRA-GA results entry by entry, including the standard deviations. Thus, the table places LoRA-Pro's reported Full FT results alongside LoRA and LoRA-GA results carried over from the LoRA-GA paper, rather than rerunning every method through one shared evaluation code path. The public evaluation scripts also use different generation and sampling settings.

This pattern propagates through later comparisons. GoRA explicitly adopts its CLIP baseline from LoRA-Pro, and CDKA uses the non-starred GoRA results as inherited baselines. When later studies reuse these published numbers, the original training and evaluation conditions are carried forward. Comparisons of a new adapter update rule therefore require aligned source code, training protocol, tuning budget, checkpoint selection, and evaluation path so that the resulting performance difference can be attributed to the update rule itself.

Our contribution addresses this attribution problem at a controlled level. At a fixed source state, we hold the model parameters, optimizer state, RNG semantics, source batch, candidate menu, and evaluation rule fixed, and vary only the selection evidence and candidate choice. This isolates the local utility of update selection from differences in prompt, masking, batching, checkpoint, and evaluation.

Overall clarification and latest endpoint results

The paper studies a controlled question: at a fixed LoRA state, can task-relevant evidence select a more useful next update from a declared candidate set than the current training batch? The fixed-state protocol separates candidate construction, selection evidence, and one-step evaluation, making the ranking and utility of candidate updates directly measurable.

The latest endpoint experiments use the same Qwen3-4B-Base / CSQA / seed 47 / rank 8 / q-v / 819-step protocol for all three adapter settings:

MethodBaseline QBest tested variantBest QChange
LoRA / Full1031/1221 (84.439%)independent weak tie term, λ=.0051038/1221 (85.012%)+7, +0.573 pp
LoRA+1046/1221 (85.667%)per-row near-boundary conflict projection1049/1221 (85.913%)+3, +0.246 pp
AnLR1032/1221 (84.521%)independent weak tie term, λ=.0051040/1221 (85.176%)+8, +0.655 pp

These results show that task-relevant evidence changes candidate ranking and mean one-step utility at a fixed source state, and that the resulting update choices can change endpoint Accuracy across LoRA, LoRA+, and AnLR. Together with the provenance audit, they support a direct methodological message: update-selection effects and adapter-method effects become interpretable when the source state, candidate set, tuning budget, checkpoint rule, and evaluation path are shared.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.