- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes GramLoRA, a cross-scale optimization trajectory transfer framework designed to improve the parameter-efficient fine-tuning (specifically LoRA) of large language models. GramLoRA leverages the optimization trajectory of a fully fine-tuned, smaller same-family model to guide target-model training. The method utilizes truncated SVD and random QR-derived orthogonal projection to map the source model's intermediate parameter residuals. The final checkpoint provides function-preserving initialization, while intermediate checkpoints supply stage-dependent gradient guidance. Experiments show improvements under limited target-model training budgets across several benchmarks.
Critical strengths
- The paper addresses a practically important problem. Reusing task information learned by smaller models could improve large-model adaptation under limited training budgets.
- The paper explores a promising direction for PEFT by transferring intermediate optimization states in addition to the final adapted weights.
- The proposed method outperforms other baselines like PiSSA, LoRA-GA, and HiRA under limited target-model training budgets across three major LLM families.
Critical weaknesses
- The empirical advantages of GramLoRA rapidly disappear and even become negative when the target model is trained with larger budgets (e.g., full dataset for 3 epochs), where baseline methods like HiRA achieve higher average accuracy.
- The paper uses QR-derived random orthonormal bases to project source singular vectors to target dimensions. It is empirically effective but lacks theoretical explanation regarding why completely random projections preserve meaningful semantic gradient directions across different parameter spaces. The benefit of random cross-model projection is not clear.
- Matching checkpoints by training progress lacks direct justification. Small and large models may learn at different rates, so equal fractions of training may not correspond to comparable learning stages. Alternative checkpoint-selection rules are not evaluated.
- Although the target model converges faster or better under a limited budget, the requirement to fully fine-tune a smaller source model on the exact same downstream datasets introduces a non-trivial offline computational burden, which leads to a hidden computational overhead. The comparison to baselines excludes the additional cost of source training.
Questions for authors
Please address the concerns raised in the Weakness part above.
4 comments
Reponse to Reviewer T6mL R2
W3: Justification and Evaluation of Checkpoint Matching Strategies
R3: Thank you for raising this important concern. We agree that equal normalized training progress does not necessarily imply equivalent learning states between source and target models, as models of different scales may exhibit different learning dynamics.
Our use of normalized training progress in Eq. (9) is intended as a simple, calibration-free checkpoint selection strategy, rather than an assumption that source and target models learn at identical rates. The fixed-final-state and shuffled-trajectory comparisons in Table 3 provide empirical evidence that stage-aware guidance is beneficial, but do not establish that normalized progress is the optimal matching criterion.
Additional experiments. To further investigate this issue, we evaluated two alternative checkpoint selection strategies on Qwen2.5-7B under the same limited training budget (20% of the training data for 0.5 epoch):
-
Validation-loss matching: We compare the normalized validation-loss reductions of the source and target models on the same validation data and select the source checkpoint with the closest normalized loss progress. The reductions are normalized using the initial and reference losses. The source reference corresponds to its final checkpoint, while the target reference is obtained from a separately available target adapter. Thus, this strategy requires additional target-side calibration information.
-
Gradient-direction matching: We estimate the target LoRA task gradients on a fixed validation sample and select the source checkpoint whose mapped trajectory-guidance gradients have the highest cosine similarity with the target task gradients in the LoRA factor space, following the gradient definitions in Eqs. (11)–(12).
All experiments use the same source trajectory, trajectory-informed initialization, and guidance-strength settings, varying only the checkpoint selection rule.
| Checkpoint Selection Rule | Eight-task Avg. Acc. |
|---|---|
| Normalized training progress (GramLoRA) | 86.90% |
| Validation-loss matching | 84.62% |
| Gradient-direction matching | 84.95% |
The normalized-progress strategy outperforms validation-loss and gradient-direction matching by 2.28 and 1.95 percentage points, respectively. These results support the effectiveness of normalized-progress matching in the evaluated setting, without requiring additional target-side calibration.
We acknowledge that source and target models may follow different learning dynamics, and these results do not establish normalized
W4: Computational Overhead of Source-Model Fine-Tuning
R4: Thank you for raising this important concern. We agree that source-model preparation introduces additional computational cost, which should be accounted for when evaluating GramLoRA's training efficiency. In our framework, source-model training and trajectory construction are performed offline, while target-model adaptation retains the standard LoRA parameterization without introducing additional trainable parameters.
To provide a more comprehensive assessment of computational cost, we measured the time required for source-model preparation and target-model adaptation. Source-model preparation includes full-parameter fine-tuning and trajectory construction. All methods use the same target-model training setting (20% of the training data for 0.5 epoch). The results on Qwen2.5-7B are summarized below.
| Method | Source Preparation | Target Training | Total Measured Time | Avg. Acc. |
|---|---|---|---|---|
| LoRA | — | 71m 11s | 71m 11s | 78.4% |
| DoRA | — | 81m 50s | 81m 50s | 82.7% |
| HiRA | — | 90m 24s | 90m 24s | 84.7% |
| GramLoRA | 8m 53s | 83m 11s | 92m 4s | 86.9% |
When source-model preparation is included, GramLoRA achieves the highest eight-task average accuracy with a comparable total measured time. Compared with HiRA (84.7% in 90m 24s), GramLoRA achieves 86.9% in 92m 4s, improving accuracy by 2.2 percentage points with only 1m 40s (approximately 1.8%) additional time.
These results show that GramLoRA provides a favorable accuracy–time trade-off under limited training budgets, even after accounting for the additional source-model preparation cost. Although the total measured times are not exactly matched, the comparison accounts for both source preparation and target adaptation, providing a more comprehensive assessment of the computational overhead.
We will include this computational cost analysis in the revised manuscript and explicitly distinguish source-model preparation from target-model adaptation costs.
I appreciate authors' effort in running additional experiments and providing further clarifications. Since the improvement from Random QR Projection and Checkpoint Matching Strategies are limited, I choose to maintain my score and leave the final decision to the AC.
We sincerely thank the reviewer for reconsidering our responses and additional experiments. We respect your decision and appreciate your feedback.
We would like to clarify one point: the QR projection and checkpoint matching experiments are intended to validate specific design choices rather than serve as the primary contributions of GramLoRA. Our central contribution lies in transferring optimization trajectories from smaller source models to guide the initialization and training of larger target models under limited training budgets.
We hope this distinction, together with the main experimental results, helps clarify the overall contribution of our work. Thank you again for your time and consideration.
Reponse to Reviewer T6mL
We sincerely thank the reviewer for the careful evaluation and constructive feedback. We appreciate the valuable comments and suggestions. Below, we address each concern point by point.
W1: Performance under Different Training Budgets
R1: Thank you for highlighting this observation. We emphasize that the primary goal of GramLoRA is to improve adaptation efficiency under limited target-model training budgets, rather than to maximize final performance after extensive training.
GramLoRA retains the standard LoRA parameterization and leverages the fine-tuning trajectory of a smaller source model to guide target-model adaptation. In contrast, HiRA modifies the adaptation parameterization. Thus, the two methods improve PEFT from different perspectives: optimization guidance versus adaptation parameterization.
As shown in Figure 2, GramLoRA achieves its largest advantage when the target model is trained on 20% of the training data for 0.5 epoch. Its relative advantage decreases as the training budget increases, and HiRA achieves higher average accuracy after three full epochs. This trend suggests that transferred optimization information is particularly beneficial during early adaptation, while its relative contribution diminishes as target-model training progresses.
These results clarify the intended application of GramLoRA: resource-constrained settings where high accuracy must be achieved with limited target-model training. We will emphasize this scope and its practical implications in the revised manuscript.
W2: Theoretical Motivation and Effectiveness of Random QR Projection
R2: Thank you for raising this important concern. We agree that the motivation for using random QR-derived projections deserves further clarification.
Motivation from prior work. Previous studies have demonstrated the effectiveness of random projections in model adaptation. [1] show that language models can be effectively fine-tuned in randomly projected low-dimensional parameter spaces. Similarly, [2] demonstrates that fixed random matrices can serve as effective components of parameter-efficient adaptation. These findings provide methodological motivation for using random projections, although they do not directly establish semantic alignment across different model parameter spaces.
Motivation for QR projection in GramLoRA. Unlike methods that learn adaptations within random subspaces, GramLoRA transfers task-relevant optimization information extracted from the fine-tuning trajectory of a smaller source model. Specifically, we first apply truncated SVD to the source-model parameter residuals to extract their dominant low-rank components. We then use fixed QR-derived orthonormal projections to map these components into the target-model dimensions.
The motivation for using QR-derived projections is to provide a geometrically structured mapping. Orthonormal mappings preserve inner products and angles where they act isometrically, although dimensionality reduction cannot guarantee these properties over the entire source space. Moreover, the projection matrices remain fixed across trajectory stages for each source–target shape pair, avoiding changes in the projection mapping across training stages.
Importantly, we do not claim that random projections guarantee semantic alignment between different models. Instead, the transferred trajectory serves as an optimization prior, whose effectiveness depends on the compatibility between source and target models.
Empirical validation. To evaluate the effect of projection construction, we conducted additional experiments on Qwen2.5-7B under the same limited training budget (20% of the training data for 0.5 epoch), keeping all other components unchanged. The results are summarized below:
The QR-derived projection achieves the highest average accuracy among the evaluated strategies, outperforming Gaussian and coordinate projections by 0.94 and 1.86 percentage points, respectively. Furthermore, Figure 3 shows that same-family trajectory transfer consistently outperforms the tested cross-family transfers, suggesting that transfer effectiveness depends on source–target compatibility rather than random projection alone.
Overall, prior work provides methodological motivation for using fixed random projections, while our experiments provide empirical support for the QR-derived projection strategy in cross-model trajectory transfer. We will clarify this motivation, its theoretical limitations, and the additional experimental results in the revised manuscript.
References
[1] Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. Proceedings of ACL-IJCNLP 2021, pp. 7319–7328.
[2] VeRA: Vector-based Random Matrix Adaptation. ICLR 2024.