acceptodds
Withdrawn from ICLR 2027

BoostLoRA: Growing Effective Rank by Boosting Adapters

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Parameter-efficient fine-tuning (PEFT) methods face a tradeoff between adapter size and expressivity: ultra-low-parameter adapters are confined to fixed low-rank subspaces, capping performance even with extended training. We propose BoostLoRA, a gradient-boosting framework that overcomes this limit by iteratively training and merging minimal adapters on the examples the current model gets wrong. A ROTATE SVD basis strategy assigns each round to an orthogonal subspace, so cumulative effective rank grows linearly with the number of rounds while each adapter remains ultra-low-rank. After merging, adapters are discarded, leaving zero inference overhead. On Qwen2.5-3B, BoostLoRA reaches 89.1% on GSM8K and 68.8% on MATH-500, surpassing both the best single-shot ultra-low-parameter adapter (TinyLoRA) and full fine-tuning; on code generation it reaches 57.2% on MBPP and 80.4% on HumanEval while full fine-tuning drops below the zero-shot baseline. We also demonstrate general applicability across domains via protein binding classification with ESM2-650M and cross-entropy training. BoostLoRA is, to our knowledge, the first ultra-low PEFT method whose effective rank grows with training, separating per-round parameter cost from total representational capacity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
parameter-efficient fine-tuning, low-rank adaptation, LoRA, gradient boosting, effective rank, model merging, large language models, reinforcement learning for reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YyQdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

BoostLoRA sequentially trains and merges TinyLoRA adapters to accumulate higher-rank weight updates while keeping the peak number of trainable adapter coefficients extremely small. Each round identifies incorrect training examples, trains a new adapter on them using GRPO or cross-entropy, and merges the update into the backbone. ROTATE selects disjoint left and right singular-vector blocks from a fixed SVD of the original pretrained weights, allowing cumulative rank rT under the stated conditions. The default setup trains 12 adapter coefficients per round and 240 coordinates across 20 rounds. The paper provides rank-growth and generalization analyses and evaluates math and code tasks with Qwen2.5-3B-Instruct and protein classification with ESM2-650M, supported by basis ablations, spectral measurements, and CNN experiments.

Critical strengths
  1. The problem is meaningful. Expanding the cumulative update space under a very small peak trainable state is a worthwhile tradeoff in parameter-efficient adaptation, with potential practical value.

  2. ROTATE offers a clear additional design. Assigning successive rounds to disjoint blocks of a fixed SVD expands the cumulative update directions while training very few coefficients per round, and yields an explicit rank-growth result under the stated conditions.

  3. The central basis strategy has meaningful empirical support. Table 3 compares TOP and ROTATE with the same round count and per-round coefficient budget, reporting gains of 1.4 and 1.5 points on GSM8K and MATH-500. Figure 3 also verifies cumulative effective-rank growth. Together, these results directly support a local benefit from switching directions.

Critical weaknesses
  1. The related-method comparisons and protocols do not establish overall superiority. The main table omits standard LoRA and LoRA-XS. Math TinyLoRA and full-finetuning results come from Morris et al. (2026), but their training data, evaluation, and checkpoint selection are not clearly aligned with the reported BoostLoRA results. CoLA is included in an appendix, but its approximately 5–8% GSM8K Instruct scores are unexpectedly far below the 76% starting point in the main table. Comparability of adapted modules and training objectives is also unclear. Some settings are reported, but no inspectable implementation is provided in the reviewed materials, limiting verification.

  2. Additional training and cumulative degrees of freedom remain plausible explanations. BoostLoRA runs 20 rounds and trains 240 cumulative coordinates, without comparing against longer-trained TinyLoRA under matched rollout or compute budgets. Reporting its own runtime does not replace baseline memory/time/performance comparisons. The single-round rank-40 ablation trains only 12 coefficients and therefore does not match cumulative degrees of freedom. The central failure-set design also lacks a full-data or equally sized random-subset control.

  3. Several results require consistency checks. For standard binary F1 and accuracy computed from the same predictions, F1 ≤ 2a/(1+a). BoostLoRA 129K in Table 2 has 58.2% accuracy, implying an F1 upper bound of approximately 73.58%, yet reports 79.2%; other configurations raise similar concerns. The GSM8K curve in Figure 1 ends around 86.2%, while the legend and text report 89.1%. HumanEval reports the cross-round peak of 80.4%, without clearly specifying independent validation-based round selection. Raw counts and explicit mappings between experiments are needed.

  4. The theoretical and motivational claims exceed the established evidence. The conditional rank-growth result in Theorem 1 is supported, but does not itself establish task capacity or generalization benefits. Theorem 2 treats evolving, data-dependent features as fixed without sufficient control of linearization error. Deriving a T-independent feature-norm bound and tighter complexity from SVD-block orthogonality also requires further justification. The introduction attributes TinyLoRA’s performance plateau to its fixed subspace without adequately excluding optimization or training-budget explanations. The abstract’s full-finetuning comparison should also be qualified: MATH-500 is 68.8%, versus 69.0% for full finetuning.

Questions for authors
  1. Please map each main result to its source and configuration, including training data, backbone checkpoint, generation/evaluation settings, failure-set construction, and checkpoint selection. Do Table 1’s 89.1% and Figure 1 come from the same SimpleRL run? How is the curve/legend discrepancy explained? Was HumanEval round selection based on an independent validation set?

  2. Can you provide standard LoRA, LoRA-XS, and a reliable iterative-merging baseline under aligned protocols? In particular, what causes the unusually low CoLA GSM8K Instruct scores, and were its training objective and evaluation implementation consistent with BoostLoRA?

  3. How does BoostLoRA compare with longer-trained TinyLoRA at matched rollout-token or GPU-time budgets? Please provide actual peak memory and runtime comparisons, and a failure-set versus full-data control to establish the source of the gains.

  4. Please provide TP, FP, TN, and FN for Table 2, together with the F1 definition, thresholds, and aggregation procedure. How can 58.2% accuracy and 79.2% F1 hold simultaneously for BoostLoRA 129K?

  5. How does Theorem 2 account for feature drift, data dependence, and linearization error across rounds? What additional assumptions establish ROTATE’s complexity improvement? If the result applies only to an idealized model, please explicitly limit the theorem and accompanying interpretation to that setting.

Reviewer_26E9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors propose BoostLoRA, which fine-tunes a model over a sequence of rounds by first finding the training examples that the current model gets wrong and train a new ultra-low-parameter LoRA-style adapter on those failures only. They then merge the adapter into the weights and repeat. Each adapter acts in a low-rank subspace spanned by singular vectors of the weight matrix. Rotating that subspace across rounds lets the rank of the merged update grow with the number of rounds. The authors validate the method by showing a rank-growth result and empirical evaluation on math reasoning and code generation with GRPO, and on protein-binding classification.

Critical strengths
  • The idea is simple and easy to implement. Each round trains only a handful of parameters, yet the merged update keeps gaining rank across rounds. Because the adapters are merged into the weights, inference costs nothing extra.
  • The TOP vs ROTATE comparison (Fig. 2, Fig. 3) cleanly shows that a fixed low-rank subspace saturates while rotating subspaces keep improving.
  • Good empirical results on GSM8k and Math500 and experiments involving domains beyond language.
Critical weaknesses
  • The method is a combination of various existing methods : sequential train-and-merge adapters (Chain of LoRA, ReLoRA, XGBLoRA), rank growth via subspace switching (GaLore, Flora; not cited), and the 12-parameter adapter (TinyLoRA). XGBLoRA, ReLoRA and other ultra low PEFT methods are not compared, and the "first ultra-low PEFT method whose effective rank grows" claim is overstated. There is no ablation running the same ROTATE schedule on the full training set.

  • Results appear to be single runs, often reported at their peak round. With no validation split for GSM8K or MATH, the stopping rule and reported rounds seem to be chosen on test accuracy. The base model scores well below official Qwen2.5-3B-Instruct numbers (GSM8K 76.0 vs 86.7, MBPP 49.8 vs 72.7), so part of the gain may be format learning. Only one LLM and two math benchmarks are used. Harder benchmarks (Minerva, OlympiadBench, AIME24, AMC23) are omitted.

  • The math baselines are 3-epoch GRPO run baselines from Morris et al., not compute-matched to 20 rounds (184–258 hours on 8 GPUs). The code full-FT baseline's algorithm is unstated, but its HumanEval drop is attributed to SFT; if so, the comparison is GRPO vs. SFT, which is not a fair comparison. CoLA (5.9% on GSM8K) and code TinyLoRA (below the base model on HumanEval) look broken.

  • Theory is weak. Theorem 1 is elementary. Theorem 2 is also textbook and relies on a linearization whose feature maps depend on earlier rounds' data, and it is checked only on a small CIFAR-10 CNN. The bound seems to be supported by adapter norms decaying over rounds (Fig. 4), yet App. D states that on Qwen2.5-3B the norms are "more uniform across rounds" so these two claims are inconsistent.

[1] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection Zhao et al. ICML 2024

[2] Flora: Low-Rank Adapters Are Secretly Gradient Compressors Hao et al. ICML 2024

Questions for authors
  • How many seeds were run? Was a validation split held out from the training data for the stopping rule and round selection, or was test accuracy used?
  • Do these results hold on the harder benchmarks?
  • What do a compute-matched TinyLoRA/LoRA baseline (same total GRPO steps) get?
  • Was the code full-FT baseline trained with SFT or GRPO?
  • How is the failure set FtF_t built: is M(x)M(x) a greedy decode or a sampled completion, and at what temperature?
  • What happens without failure filtering, i.e. the same 20-round ROTATE schedule on the full training set?
Reviewer_ePogOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an iterative boosting-like mechanism for parameter-efficient fine-tuning (PEFT). The paper focuses specifically on PEFT scenarios with a very small number of trainable parameters and builds on the recent TinyLoRA work to that end. Specifically, in each gradient-boosting round, the approach first evaluates the entire training set and determines failure cases. A TinyLoRA adapter is then trained on the failure cases and merged with the base model / previous rounds. Two different variants are considered: The first variant adapts in the subspace of the top-r singular vectors (of the current weight matrix, which can change over boosting rounds) while the second variant operates in sequential orthogonal subspaces of the SVD of the original weight matrix. Application scenarios consider large language models and protein binding classification. Experiments show that even with a very limited parameter budget, boosting TinyLoRA adapters can work substantially better than a single TinyLoRA adapter with a comparable parameter budget. The experiments also find the proposed approach to outperform full fine-tuning despite the small parameter budget.

Critical strengths
  • While there has been prior work that combines LoRA with gradient boosting, this seems to be the first work that does so in a regime of very few trainable parameters ("ultra-low-rank") and applies this in the context of RL.
  • The paper is mostly well-written and found to clearly outperform the TinyLoRA baseline.
Critical weaknesses
  • The novelty of the paper is somewhat limited. Gradient boosting has been combined with LoRA before (XGBLoRA, cited). The main difference is thus two-fold: (1) the present paper uses TinyLoRA rather than rank-1 LoRA as weak learner, and (2) the paper complements cross-entropy based fine-tuning with RL-based fine-tuning. The jury on how important these differences are is still out. As far as I could tell, the TinyLoRA approach used as weak learner has not yet been published in a peer-reviewed venue. If TinyLoRA turns out to be impactful (not clear yet, AFAIK), then this approach may prove to be impactful as well, even if it is technically not particularly novel or innovative. Moreover, lacking detailed comparisons against combinations of previous multi-round fine-tuning approaches with RL (see below), we do not gain much insight into the benefits over previous approaches other than the TinyLoRA weak learner itself.
  • The choice of baselines and base-cases in the experimental comparison is very limited. Essentially, the paper compares the proposed approach to a single-run of TinyLoRA. We can conclude that it works better than a single run of TinyLoRA, but we do not gain much insight into broader comparisons with other multi-round fine-tuning approaches. Yes, XGBLoRA and CoLA were originally applied in different settings, but it should be possible to use them also with RL-based fine-tuning. The main paper has no comparisons with such previous approaches. Only appendix E contains a comparison with CoLA, but the details of the comparison remain unclear. Is this the same RL-setup as for the proposed approach, only replacing boosted TinyLoRAs with CoLA? Some other experimental weaknesses:
    • Only rank-2 updates are considered. What is the motivation for this and how does this generalize to other ranks?
    • Table 1 does not do an apples-to-apples comparison. TinyLoRA uses 16 parameters, BoostLoRA 20 rounds of 12 parameters. I could sort of understand 12 vs. 12x40 parameters (even if the presentation in Table 1 quoting parameters updated simultaneously is quite misleading), but why 16 vs. 12 (or 240)? The ablation study in Table 3 clarifies things a bit, i.e., here we see a comparison of BoostLoRA with 12x40 parameters vs. TinyLoRA with 240 parameters, which is absent from Table 1.
    • As said, TinyLoRA and full fine-tuning are the only baselines. This is not enough.
    • The protein binding classification setting (Table 2) is also a bit contrived. I laud the authors for using a different setting, specifically one that does not use RL, but according to the authors, "No prior classification baselines exist for this dataset" (l. 381f). A setting where prior results exists would be much more convincing. Also, again only Tiny-LoRA, full fine-tuning, and a linear probe are compared, no other PEFT approaches.
  • There are minor technical weaknesses: l. 235 claims that all rounds of the TOP variant (adaptation of top-singular vectors) use the same columns. But this is not completely true as far as I can see. As discussed in l. 188, this is only approximately true, since the weight matrix and thus the singular vectors change only slightly from round to round.

Smaller points:

  • There are several notational weaknesses/inconsistencies (see questions below for details).
  • While CoLA is experimentally compared, the related work or the technical section does not discuss the technical differences.
  • The repeated claim of zero inference overhead is a bit misleading here. This is typical of LoRA-based approaches in general and not a particular benefit of the proposed approach. Similarly, the claim of zero additional parameters after merging (l. 331) is misleading.
  • The paper makes a big point out of the proposed approach outperforming full fine-tuning. That's of course good as a reference, but this cannot necessarily be taken as a strong assurance that the proposed approach is working well. Outperforming full fine-tuning has been a regularly observed phenomenon in PEFT (of course, this depends a bit on the setting).
  • The determination of the set of failure cases induces an inherent limitation of the approach: it requires binary failure cases and is thus only directly applicable to classification tasks and related settings. This limitation is not clearly discussed.
  • Some language is very "un-human", e.g. l. 242 ("Define a model error...") or l. 248 ("Write..."), and sounds more like a prompt for an LLM.
Questions for authors
  • How is R_t in Sec. 3.2, step 3 "Merge" defined? What is the to R(vt)R(v_t) in line 9 of Algorithm 1? The R(⋅)R(\cdot) notation is not defined. Why use the ∑tΔt\sum_t \Delta_t notation (l. 186) if you have defined Δˉt\bar{\Delta}_t?
  • What is the definition and meaning of rir_i and σr\sigma_r in Eq. 3?
  • How well does BoostLoRA compare to other multi-round PEFT methods in the same RL setting? Why do you consider the comparison to TinyLoRA and full fine-tuning to be sufficient?

Smaller questions:

  • Is it really a surprise that ROTATE works better than TOP (l. 411)? After all, ROTATE works in orthogonal subspaces, which allow for a broader adaptation.
  • Why is BoostLoRA with T=1in Table 3 not the same as TinyLoRA with r=40?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.