- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
BoostLoRA sequentially trains and merges TinyLoRA adapters to accumulate higher-rank weight updates while keeping the peak number of trainable adapter coefficients extremely small. Each round identifies incorrect training examples, trains a new adapter on them using GRPO or cross-entropy, and merges the update into the backbone. ROTATE selects disjoint left and right singular-vector blocks from a fixed SVD of the original pretrained weights, allowing cumulative rank rT under the stated conditions. The default setup trains 12 adapter coefficients per round and 240 coordinates across 20 rounds. The paper provides rank-growth and generalization analyses and evaluates math and code tasks with Qwen2.5-3B-Instruct and protein classification with ESM2-650M, supported by basis ablations, spectral measurements, and CNN experiments.
Critical strengths
-
The problem is meaningful. Expanding the cumulative update space under a very small peak trainable state is a worthwhile tradeoff in parameter-efficient adaptation, with potential practical value.
-
ROTATE offers a clear additional design. Assigning successive rounds to disjoint blocks of a fixed SVD expands the cumulative update directions while training very few coefficients per round, and yields an explicit rank-growth result under the stated conditions.
-
The central basis strategy has meaningful empirical support. Table 3 compares TOP and ROTATE with the same round count and per-round coefficient budget, reporting gains of 1.4 and 1.5 points on GSM8K and MATH-500. Figure 3 also verifies cumulative effective-rank growth. Together, these results directly support a local benefit from switching directions.
Critical weaknesses
-
The related-method comparisons and protocols do not establish overall superiority. The main table omits standard LoRA and LoRA-XS. Math TinyLoRA and full-finetuning results come from Morris et al. (2026), but their training data, evaluation, and checkpoint selection are not clearly aligned with the reported BoostLoRA results. CoLA is included in an appendix, but its approximately 5–8% GSM8K Instruct scores are unexpectedly far below the 76% starting point in the main table. Comparability of adapted modules and training objectives is also unclear. Some settings are reported, but no inspectable implementation is provided in the reviewed materials, limiting verification.
-
Additional training and cumulative degrees of freedom remain plausible explanations. BoostLoRA runs 20 rounds and trains 240 cumulative coordinates, without comparing against longer-trained TinyLoRA under matched rollout or compute budgets. Reporting its own runtime does not replace baseline memory/time/performance comparisons. The single-round rank-40 ablation trains only 12 coefficients and therefore does not match cumulative degrees of freedom. The central failure-set design also lacks a full-data or equally sized random-subset control.
-
Several results require consistency checks. For standard binary F1 and accuracy computed from the same predictions, F1 ≤ 2a/(1+a). BoostLoRA 129K in Table 2 has 58.2% accuracy, implying an F1 upper bound of approximately 73.58%, yet reports 79.2%; other configurations raise similar concerns. The GSM8K curve in Figure 1 ends around 86.2%, while the legend and text report 89.1%. HumanEval reports the cross-round peak of 80.4%, without clearly specifying independent validation-based round selection. Raw counts and explicit mappings between experiments are needed.
-
The theoretical and motivational claims exceed the established evidence. The conditional rank-growth result in Theorem 1 is supported, but does not itself establish task capacity or generalization benefits. Theorem 2 treats evolving, data-dependent features as fixed without sufficient control of linearization error. Deriving a T-independent feature-norm bound and tighter complexity from SVD-block orthogonality also requires further justification. The introduction attributes TinyLoRA’s performance plateau to its fixed subspace without adequately excluding optimization or training-budget explanations. The abstract’s full-finetuning comparison should also be qualified: MATH-500 is 68.8%, versus 69.0% for full finetuning.
Questions for authors
-
Please map each main result to its source and configuration, including training data, backbone checkpoint, generation/evaluation settings, failure-set construction, and checkpoint selection. Do Table 1’s 89.1% and Figure 1 come from the same SimpleRL run? How is the curve/legend discrepancy explained? Was HumanEval round selection based on an independent validation set?
-
Can you provide standard LoRA, LoRA-XS, and a reliable iterative-merging baseline under aligned protocols? In particular, what causes the unusually low CoLA GSM8K Instruct scores, and were its training objective and evaluation implementation consistent with BoostLoRA?
-
How does BoostLoRA compare with longer-trained TinyLoRA at matched rollout-token or GPU-time budgets? Please provide actual peak memory and runtime comparisons, and a failure-set versus full-data control to establish the source of the gains.
-
Please provide TP, FP, TN, and FN for Table 2, together with the F1 definition, thresholds, and aggregation procedure. How can 58.2% accuracy and 79.2% F1 hold simultaneously for BoostLoRA 129K?
-
How does Theorem 2 account for feature drift, data dependence, and linearization error across rounds? What additional assumptions establish ROTATE’s complexity improvement? If the result applies only to an idealized model, please explicitly limit the theorem and accompanying interpretation to that setting.