acceptodds
Under review as a conference paper at ICLR 2027

GramLoRA: Cross-Scale Optimization Trajectory Transfer for LoRA Fine-Tuning

Abstract

Parameter-efficient fine-tuning (PEFT), particularly low-rank adaptation (LoRA), has become a standard approach for adapting large language models without updating the full parameter space. Existing LoRA variants mainly improve the parameterization, initialization, or optimization of low-rank updates, while adaptation is typically learned independently for each target model. We ask a complementary question: can the optimization trajectory of a smaller model guide LoRA adaptation of a larger one? We introduce GramLoRA, a cross-scale framework that transfers the optimization trajectory of a smaller same-family source model to guide LoRA adaptation of a larger target model. We fully fine-tune the source model and convert intermediate parameter residuals into target-compatible rank-\(r\) factors. The final source state provides a task-informed, identity-preserving initialization, while intermediate states provide stage-aware directional guidance without changing the downstream objective. Across Qwen2.5-7B, Gemma4-E4B, and Ministral-8B on eight commonsense reasoning tasks, GramLoRA consistently improves adaptation under limited target-model training. With 20% of Commonsense170K for 0.5 epoch, GramLoRA outperforms HiRA by 2.2, 4.6, and 2.9 average points, respectively, and also generalizes to natural language understanding and mathematical reasoning. Its advantage diminishes with extended target-model training, suggesting that transferred source-model optimization information is particularly effective for early target-model adaptation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Parameter-Efficient Fine-Tuning, Low-Rank Adaptation, Optimization Trajectory Transfer, Small-to-Large Transfer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_T6mLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GramLoRA, a cross-scale optimization trajectory transfer framework designed to improve the parameter-efficient fine-tuning (specifically LoRA) of large language models. GramLoRA leverages the optimization trajectory of a fully fine-tuned, smaller same-family model to guide target-model training. The method utilizes truncated SVD and random QR-derived orthogonal projection to map the source model's intermediate parameter residuals. The final checkpoint provides function-preserving initialization, while intermediate checkpoints supply stage-dependent gradient guidance. Experiments show improvements under limited target-model training budgets across several benchmarks.

Critical strengths
  1. The paper addresses a practically important problem. Reusing task information learned by smaller models could improve large-model adaptation under limited training budgets.
  2. The paper explores a promising direction for PEFT by transferring intermediate optimization states in addition to the final adapted weights.
  3. The proposed method outperforms other baselines like PiSSA, LoRA-GA, and HiRA under limited target-model training budgets across three major LLM families.
Critical weaknesses
  1. The empirical advantages of GramLoRA rapidly disappear and even become negative when the target model is trained with larger budgets (e.g., full dataset for 3 epochs), where baseline methods like HiRA achieve higher average accuracy.
  2. The paper uses QR-derived random orthonormal bases to project source singular vectors to target dimensions. It is empirically effective but lacks theoretical explanation regarding why completely random projections preserve meaningful semantic gradient directions across different parameter spaces. The benefit of random cross-model projection is not clear.
  3. Matching checkpoints by training progress lacks direct justification. Small and large models may learn at different rates, so equal fractions of training may not correspond to comparable learning stages. Alternative checkpoint-selection rules are not evaluated.
  4. Although the target model converges faster or better under a limited budget, the requirement to fully fine-tune a smaller source model on the exact same downstream datasets introduces a non-trivial offline computational burden, which leads to a hidden computational overhead. The comparison to baselines excludes the additional cost of source training.
Questions for authors

Please address the concerns raised in the Weakness part above.

4 comments
AuthorsOpenReview ↗

Reponse to Reviewer T6mL

We sincerely thank the reviewer for the careful evaluation and constructive feedback. We appreciate the valuable comments and suggestions. Below, we address each concern point by point.

W1: Performance under Different Training Budgets

R1: Thank you for highlighting this observation. We emphasize that the primary goal of GramLoRA is to improve adaptation efficiency under limited target-model training budgets, rather than to maximize final performance after extensive training.

GramLoRA retains the standard LoRA parameterization and leverages the fine-tuning trajectory of a smaller source model to guide target-model adaptation. In contrast, HiRA modifies the adaptation parameterization. Thus, the two methods improve PEFT from different perspectives: optimization guidance versus adaptation parameterization.

As shown in Figure 2, GramLoRA achieves its largest advantage when the target model is trained on 20% of the training data for 0.5 epoch. Its relative advantage decreases as the training budget increases, and HiRA achieves higher average accuracy after three full epochs. This trend suggests that transferred optimization information is particularly beneficial during early adaptation, while its relative contribution diminishes as target-model training progresses.

These results clarify the intended application of GramLoRA: resource-constrained settings where high accuracy must be achieved with limited target-model training. We will emphasize this scope and its practical implications in the revised manuscript.

W2: Theoretical Motivation and Effectiveness of Random QR Projection

R2: Thank you for raising this important concern. We agree that the motivation for using random QR-derived projections deserves further clarification.

Motivation from prior work. Previous studies have demonstrated the effectiveness of random projections in model adaptation. [1] show that language models can be effectively fine-tuned in randomly projected low-dimensional parameter spaces. Similarly, [2] demonstrates that fixed random matrices can serve as effective components of parameter-efficient adaptation. These findings provide methodological motivation for using random projections, although they do not directly establish semantic alignment across different model parameter spaces.

Motivation for QR projection in GramLoRA. Unlike methods that learn adaptations within random subspaces, GramLoRA transfers task-relevant optimization information extracted from the fine-tuning trajectory of a smaller source model. Specifically, we first apply truncated SVD to the source-model parameter residuals to extract their dominant low-rank components. We then use fixed QR-derived orthonormal projections to map these components into the target-model dimensions.

The motivation for using QR-derived projections is to provide a geometrically structured mapping. Orthonormal mappings preserve inner products and angles where they act isometrically, although dimensionality reduction cannot guarantee these properties over the entire source space. Moreover, the projection matrices remain fixed across trajectory stages for each source–target shape pair, avoiding changes in the projection mapping across training stages.

Importantly, we do not claim that random projections guarantee semantic alignment between different models. Instead, the transferred trajectory serves as an optimization prior, whose effectiveness depends on the compatibility between source and target models.

Empirical validation. To evaluate the effect of projection construction, we conducted additional experiments on Qwen2.5-7B under the same limited training budget (20% of the training data for 0.5 epoch), keeping all other components unchanged. The results are summarized below:

Projection MethodEight-task Avg. Acc.
QR-derived projection (GramLoRA)86.90%
Fixed Gaussian projection85.96%
Coordinate projection85.04%

The QR-derived projection achieves the highest average accuracy among the evaluated strategies, outperforming Gaussian and coordinate projections by 0.94 and 1.86 percentage points, respectively. Furthermore, Figure 3 shows that same-family trajectory transfer consistently outperforms the tested cross-family transfers, suggesting that transfer effectiveness depends on source–target compatibility rather than random projection alone.

Overall, prior work provides methodological motivation for using fixed random projections, while our experiments provide empirical support for the QR-derived projection strategy in cross-model trajectory transfer. We will clarify this motivation, its theoretical limitations, and the additional experimental results in the revised manuscript.

References

[1] Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. Proceedings of ACL-IJCNLP 2021, pp. 7319–7328.

[2] VeRA: Vector-based Random Matrix Adaptation. ICLR 2024.

AuthorsOpenReview ↗

Reponse to Reviewer T6mL R2

W3: Justification and Evaluation of Checkpoint Matching Strategies

R3: Thank you for raising this important concern. We agree that equal normalized training progress does not necessarily imply equivalent learning states between source and target models, as models of different scales may exhibit different learning dynamics.

Our use of normalized training progress in Eq. (9) is intended as a simple, calibration-free checkpoint selection strategy, rather than an assumption that source and target models learn at identical rates. The fixed-final-state and shuffled-trajectory comparisons in Table 3 provide empirical evidence that stage-aware guidance is beneficial, but do not establish that normalized progress is the optimal matching criterion.

Additional experiments. To further investigate this issue, we evaluated two alternative checkpoint selection strategies on Qwen2.5-7B under the same limited training budget (20% of the training data for 0.5 epoch):

  • Validation-loss matching: We compare the normalized validation-loss reductions of the source and target models on the same validation data and select the source checkpoint with the closest normalized loss progress. The reductions are normalized using the initial and reference losses. The source reference corresponds to its final checkpoint, while the target reference is obtained from a separately available target adapter. Thus, this strategy requires additional target-side calibration information.

  • Gradient-direction matching: We estimate the target LoRA task gradients on a fixed validation sample and select the source checkpoint whose mapped trajectory-guidance gradients have the highest cosine similarity with the target task gradients in the LoRA factor space, following the gradient definitions in Eqs. (11)–(12).

All experiments use the same source trajectory, trajectory-informed initialization, and guidance-strength settings, varying only the checkpoint selection rule.

Checkpoint Selection RuleEight-task Avg. Acc.
Normalized training progress (GramLoRA)86.90%
Validation-loss matching84.62%
Gradient-direction matching84.95%

The normalized-progress strategy outperforms validation-loss and gradient-direction matching by 2.28 and 1.95 percentage points, respectively. These results support the effectiveness of normalized-progress matching in the evaluated setting, without requiring additional target-side calibration.

We acknowledge that source and target models may follow different learning dynamics, and these results do not establish normalized

W4: Computational Overhead of Source-Model Fine-Tuning

R4: Thank you for raising this important concern. We agree that source-model preparation introduces additional computational cost, which should be accounted for when evaluating GramLoRA's training efficiency. In our framework, source-model training and trajectory construction are performed offline, while target-model adaptation retains the standard LoRA parameterization without introducing additional trainable parameters.

To provide a more comprehensive assessment of computational cost, we measured the time required for source-model preparation and target-model adaptation. Source-model preparation includes full-parameter fine-tuning and trajectory construction. All methods use the same target-model training setting (20% of the training data for 0.5 epoch). The results on Qwen2.5-7B are summarized below.

MethodSource PreparationTarget TrainingTotal Measured TimeAvg. Acc.
LoRA—71m 11s71m 11s78.4%
DoRA—81m 50s81m 50s82.7%
HiRA—90m 24s90m 24s84.7%
GramLoRA8m 53s83m 11s92m 4s86.9%

When source-model preparation is included, GramLoRA achieves the highest eight-task average accuracy with a comparable total measured time. Compared with HiRA (84.7% in 90m 24s), GramLoRA achieves 86.9% in 92m 4s, improving accuracy by 2.2 percentage points with only 1m 40s (approximately 1.8%) additional time.

These results show that GramLoRA provides a favorable accuracy–time trade-off under limited training budgets, even after accounting for the additional source-model preparation cost. Although the total measured times are not exactly matched, the comparison accounts for both source preparation and target adaptation, providing a more comprehensive assessment of the computational overhead.

We will include this computational cost analysis in the revised manuscript and explicitly distinguish source-model preparation from target-model adaptation costs.

I appreciate authors' effort in running additional experiments and providing further clarifications. Since the improvement from Random QR Projection and Checkpoint Matching Strategies are limited, I choose to maintain my score and leave the final decision to the AC.

We sincerely thank the reviewer for reconsidering our responses and additional experiments. We respect your decision and appreciate your feedback.

We would like to clarify one point: the QR projection and checkpoint matching experiments are intended to validate specific design choices rather than serve as the primary contributions of GramLoRA. Our central contribution lies in transferring optimization trajectories from smaller source models to guide the initialization and training of larger target models under limited training budgets.

We hope this distinction, together with the main experimental results, helps clarify the overall contribution of our work. Thank you again for your time and consideration.

Reviewer_uWBDOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes GramLoRA, which uses the full fine-tuning of a smaller model in the same model family to guide the PEFT for the larger model. It first trains a smaller model on the same task and then uses the low-rank approximation of the final weight shift to construct the initialization of GramLoRA. The additional trajectory information is further used to guide the PEFT of the larger model. Experiments are conducted on various models and tasks.

Critical strengths
  1. The idea of transferring the information from the small model to the larger model is interesting and novel.
  2. The performance is improved under a short training setting using reduced data and training steps.
Critical weaknesses
  1. The gain of GramLoRA is limited to the short-training setting. When the model is fully fine-tuned, its performance is very close to that of other SOTA PEFT methods. In Figure 2, HiRA is consistently better than GramLoRA under the full training budget.
  2. The proposed GramLoRA is complicated, introducing the training of another model in the process, which often requires extra tuning of hyperparameters. With the extra complication, the performance gain is not substantial under the long training setting.
Questions for authors
  1. Do the choices of the projection matrices P_in and P_out affect the fine-tuning performance?
  2. How do you construct the initialization and perform trajectory guidance when the source model and target model have different numbers of layers?
  3. The performance of LoRA-GA and PiSSA are already close to GramLoRA shown in Table 1. But these two methods are not included for comparison for the Gemma and Ministral models in Table 1. It is better to include the results of these two SOTA methods for the other models.
  4. The authors claim better performance under the short training setting. However, GramLoRA has the extra cost of training the source model. So, the comparison is not fair. It would be better to compare the methods under the same time or computation budget.
3 comments
AuthorsOpenReview ↗

Response to Reviewer uWB

We sincerely thank the reviewer for taking the time to carefully evaluate our manuscript and provide constructive feedback. We appreciate the reviewer's comments and concerns, which help us clarify the motivation, methodology, and experimental evaluation of GramLoRA. Below, we address each point in detail and provide additional explanations and experimental results where appropriate.

W1: Performance under Different Training Budgets

RW1: Thank you for this observation. We emphasize that the primary goal of GramLoRA is to improve adaptation efficiency under limited target-model training budgets, rather than to maximize final performance after full training.

Importantly, GramLoRA retains the same architecture, rank, trainable parameter count, and inference procedure as standard LoRA. Its improvement comes from leveraging the fine-tuning trajectory of a smaller source model to guide target-model adaptation, rather than modifying the LoRA parameterization. In contrast, HiRA modifies the adaptation structure. Thus, GramLoRA and HiRA improve PEFT from different perspectives: optimization guidance versus adaptation parameterization.

As shown in Figure 2, GramLoRA achieves its most pronounced advantage when the target model is trained on 20% of the training data for 0.5 epoch. As the training budget increases, this advantage diminishes, and HiRA achieves higher performance under the full training budget.

Furthermore, our end-to-end training-time comparison shows that GramLoRA achieves 86.9% average accuracy in 92 min 4 s, including source-model preparation, compared with HiRA's 84.7% in 90 min 24 s. This further supports its practical value in resource-constrained adaptation settings.

We will clarify the intended scope of GramLoRA and include the end-to-end training-time comparison in the revised manuscript.

W2: Computational Complexity and Additional Training Overhead

RW2: Thank you for raising this concern. While GramLoRA introduces an additional source-model preparation stage, this stage is performed offline using a much smaller model. Importantly, GramLoRA retains the standard LoRA architecture, rank, trainable parameter count, and inference procedure, without introducing additional trainable modules into the target model.

Our additional computational cost analysis shows that GramLoRA achieves 86.9% average accuracy in 92 min 4 s, including source-model preparation, compared with HiRA's 84.7% in 90 min 24 s. Thus, GramLoRA improves accuracy by 2.2 percentage points with only approximately 1.8% additional total training time in this setting.

Regarding hyperparameter sensitivity, Figure 4 shows that GramLoRA performs consistently across the tested values of �0��0�, with the default setting achieving the best result. This suggests that the improvement does not depend on narrowly tuned guidance coefficients.

We acknowledge that the performance advantage diminishes with longer training budgets, consistent with GramLoRA's focus on efficient early-stage adaptation. We will clarify the computational overhead, hyperparameter sensitivity, and intended application scope in the revised manuscript.

Q1: Effect of Projection Matrix Construction

RQ1: Thank you for this question. To investigate the effect of the projection matrices �1��1� and �2��2�, we compare three projection strategies on Qwen2.5-7B under the same short-training setting (20% of the training data for 0.5 epoch). All other components, including trajectory-informed initialization and stage-aware trajectory guidance, remain unchanged.

Our original method constructs fixed projection matrices from orthonormal bases obtained through QR decomposition of random matrices. The same projection matrices are shared across trajectory stages for each source–target shape pair. We compare this approach with coordinate projection, which retains leading coordinates or applies zero-padding, and fixed Gaussian projection, which uses randomly sampled Gaussian matrices.

Projection MethodEight-task Avg. Acc.
QR-derived projection (GramLoRA)86.90%
Fixed Gaussian projection85.96%
Coordinate projection85.04%

The results show that the construction of �3��3� and �4��4� affects fine-tuning performance. Among the three strategies, the original QR-derived projection achieves the highest average accuracy, outperforming Gaussian and coordinate projections by 0.94 and 1.86 percentage points, respectively. We will include these results in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer uWB R2

Q2: Initialization and Trajectory Guidance Across Different Model Depths

RQ2: Thank you for raising this important question. While Section 3.1 describes how source-model residuals are projected into target-compatible dimensions, the layer correspondence between models with different numbers of Transformer layers was not sufficiently detailed. We clarify this procedure and its application to initialization and trajectory guidance below.

Layer correspondence. We adopt a depth-segment-based mapping strategy to align source and target models with different numbers of Transformer layers. Specifically, we partition the source and target layers into the same number of contiguous, depth-ordered segments. Each target layer is associated with all source layers in the corresponding segment. This correspondence remains fixed across all recorded training stages.

Segment-level residual aggregation. At each recorded source training stage, we compute the parameter residual of each source layer by subtracting its pretrained weights from its fine-tuned weights. We then average these residuals across all source layers within the same segment:

Rˉj=Rj/Nj\bar{R}_j = R_j / N_j

Here, RjR_j denotes the sum of parameter residuals across all source layers in segment jj, and NjN_j denotes the number of layers in that segment. This averaging is performed separately for each recorded training stage and relevant weight projection.

We then apply truncated SVD to the averaged residual and use the fixed projection matrices defined in Eq. (4) to construct target-compatible LoRA factors. In our Qwen implementation, target Q/K/V projections share an attention prior constructed from source Q residuals, while FFN up-projection (U) and down-projection (D) use their corresponding source projections. Thus, multiple target layers within the same segment share the prior derived from the corresponding source segment.

Trajectory-informed initialization. Following Eqs. (6)–(8), each target LoRA adapter is initialized using the transferred factors from the final source training stage (1.0). The frozen target weight is compensated by subtracting the scaled product of these initialized LoRA factors, preserving the effective pretrained weight at initialization.

Stage-aware trajectory guidance. During target-model training, the same layer correspondence is maintained. Following Eqs. (9)–(18), the active source trajectory stage is selected according to normalized target training progress. The transferred factors at that stage define a trajectory-guidance direction for each target layer. The corresponding guidance gradients are normalized and combined with downstream task gradients to update the target LoRA adapters.

Thus, initialization uses the final transferred trajectory state, while trajectory guidance uses stage-matched states, both under the same fixed layer correspondence.

Concrete example. For Qwen2.5-0.5B → Qwen2.5-7B, the source model contains 24 Transformer layers and the target model contains 28 layers. Both are partitioned into eight corresponding depth segments, with zero-based, inclusive layer indices:

SegmentSource LayersTarget Layers
00–20–3
13–54–6
26–87–9
39–1110–12
412–1413–15
515–1716–18
618–2019–21
721–2322–27

For example, target layers 0–3 use the trajectory constructed by averaging the parameter residuals of source layers 0–2 at each recorded training stage, followed by truncated SVD and dimensional projection. The final-stage factors initialize the target LoRA adapters, while the stage-matched factors guide subsequent training.

We will clarify the depth-segment mapping strategy and its application to both initialization and trajectory guidance in the revised manuscript.

Q3: Additional Comparisons with SOTA PEFT Methods

RQ3: Thank you for this valuable suggestion. We have extended the comparison by evaluating PiSSA, LoRA-GA, and LoRA-Pro on Gemma4-E4B and Ministral-8B under the same target-model training setting (20% of the training data for 0.5 epoch), consistent with Table 1.

The eight-task average accuracies are summarized below:

Target ModelPiSSALoRA-GALoRA-ProGramLoRA
Gemma4-E4B82.9%78.7%76.5%84.0%
Ministral-8B85.7%85.2%83.4%86.6%

GramLoRA achieves the highest average accuracy on both target models, outperforming the strongest additional baseline, PiSSA, by 1.1 percentage points on Gemma4-E4B and 0.9 percentage points on Ministral-8B. These results further support the effectiveness of GramLoRA across different model families under limited training budgets.

We will incorporate these comparisons and the corresponding per-task results into the revised Table 1.

AuthorsOpenReview ↗

Response to Reviewer uWB R3

Q4: Fairness of Comparison under Computational Budgets

RQ4: Thank you for raising this important concern. We agree that source-model preparation introduces additional computational cost and should be considered when evaluating GramLoRA's training efficiency. In our method, source-model training and trajectory construction are performed offline, while target-model adaptation retains the standard LoRA parameterization without introducing additional trainable parameters.

To provide a more comprehensive assessment of computational cost, we measured the time required for the complete GramLoRA training procedure, including source-model preparation and target-model adaptation. The source model is trained using full-parameter fine-tuning, as described in Section 3.1. All methods use the same target-model training setting (20% of the training data for 0.5 epoch). The results on Qwen2.5-7B are summarized below.

MethodSource PreparationTarget TrainingTotal Measured TimeAvg. Acc.
LoRA—71m 11s71m 11s78.4%
DoRA—81m 50s81m 50s82.7%
HiRA—90m 24s90m 24s84.7%
GramLoRA8m 53s83m 11s92m 4s86.9%

When source-model preparation is included, GramLoRA achieves the highest eight-task average accuracy with a comparable total measured time. Compared with HiRA (84.7% in 90m 24s), GramLoRA achieves 86.9% in 92m 4s, improving accuracy by 2.2 percentage points with only 1m 40s (approximately 1.8%) additional time.

These results demonstrate that GramLoRA provides a favorable accuracy–time trade-off under limited training budgets, even when the additional source-model preparation cost is taken into account. Although the total measured times are not exactly matched, the comparison accounts for the computational stages involved in our training procedure and provides a more comprehensive assessment of practical efficiency.

We will include this computational cost analysis in the revised manuscript to clarify the additional preparation overhead and the fairness of the comparison.

Reviewer_G71MOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

GramLoRA transfers fine-tuning information from a smaller model to a larger model in the same family. It records the smaller model’s weight changes at several training stages, compresses and maps them into low-rank updates for the larger model, then uses the final update to initialize LoRA and the intermediate updates to guide training. The larger model retains its usual task objective.

The main contribution is using a source model’s training trajectory, rather than only its final adapted weights, as guidance for target-model LoRA fine-tuning. The authors evaluate three source–target model pairs on eight commonsense reasoning tasks, comparing GramLoRA with LoRA and several LoRA variants. With 20% of the training data for 0.5 epoch, it improves the eight-task average over HiRA by 2.2 points on Qwen2.5-7B, 4.6 on Gemma4-E4B, and 2.9 on Ministral-8B. Additional experiments report gains on six GLUE tasks and GSM8K. Ablations suggest that source-informed initialization accounts for most of the improvement, with trajectory guidance adding a smaller benefit. The advantage diminishes with longer target-model training, and cross-family transfer performs worse than same-family transfer.

Critical strengths

The paper presents an interesting and potentially useful way to transfer fine-tuning information from a smaller model to a larger model. This could be valuable in practical settings where training the target model is costly or time-limited, since the smaller model may provide a helpful initialization and guidance during adaptation.

The approach also raises a broader research question: how much of one model’s training process can be transferred to another? This could make GramLoRA useful as a tool for studying and comparing models—for example, testing whether a training trajectory learned by one model can meaningfully help another. The results showing stronger transfer within the same model family make this a promising direction for further investigation.

Critical weaknesses

None

Questions for authors
  1. Could the authors clarify how they see GramLoRA relating to model similarity: does the method provide a useful way to measure whether two models share transferable training directions, and could the success or failure of trajectory transfer serve as evidence of how similar the models are? The results showing stronger transfer within the same family suggest this may be possible, but it would help to know whether the authors view this as part of the method’s intended use or as a direction for future work.

  2. How do you match source and target layers when the models have different numbers of layers? The paper explains how you project the dimensions of corresponding weight matrices, but it is unclear how you determine which source layer corresponds to each target layer. Do you match layers by their relative depth, by module type, or using another rule? Please clarify this mapping, as it seems important for reproducing the method.

3 comments
AuthorsOpenReview ↗

Response to Reviewer G71M

We sincerely thank the reviewer for the positive evaluation of our work and the constructive comments. We appreciate the reviewer's recognition of GramLoRA's contributions. Below, we provide point-by-point responses to the questions and suggestions.

Q1: Relationship between Trajectory Transfer and Model Similarity

R1: Thank you for pointing out the possible connection between trajectory transfer and model similarity. GramLoRA leverages the fine-tuning trajectory of a smaller source model to improve LoRA adaptation under a limited target-model training budget. Figure 3 and Table 13 in Appendix F show that same-family transfer achieves the highest eight-task average accuracy for each of the three target models evaluated, although cross-family transfer is also feasible with varying effectiveness.

With the target model, downstream task, mapping rule, and training budget held fixed, transfer performance across different source trajectories can provide an empirical signal of task-dependent optimization compatibility. However, this signal is directional and depends on the experimental setup. Unsuccessful transfer may also result from limitations in layer correspondence or dimensional projection, rather than fundamental model dissimilarity.

Developing a validated similarity metric based on this signal, or using it to guide source model selection before training, remains an interesting direction for future research. We will clarify this distinction in the revised manuscript.

Q2: Layer Mapping Between Source and Target Models

R2: Thank you for highlighting this important reproducibility issue. While Eq. (4) in Section 3.1 describes the dimensional projection between source and target weight matrices, the layer correspondence strategy was not sufficiently detailed in the original manuscript. We clarify this procedure below.

Layer mapping strategy. We adopt a depth-segment-based mapping strategy to align source and target models with different numbers of Transformer layers. Specifically, we partition the source and target layers into the same number of contiguous, depth-ordered segments. Let SjS_j and TjT_j denote the source and target layer sets in segment jj, respectively. Each target layer in TjT_j is associated with all source layers in SjS_j. This correspondence remains fixed across all recorded training stages, ensuring consistent transfer of the source optimization trajectory.

Segment-level residual aggregation. At each recorded source training stage, we compute the parameter residual of each source layer by subtracting its pretrained weights from its fine-tuned weights. We then average these residuals across all source layers within the same segment:

Rˉj=Rj/Nj\bar{R}_j = R_j / N_j

Here, RjR_j denotes the sum of parameter residuals across all source layers in segment jj, and NjN_j denotes the number of layers in that segment. This averaging is performed separately for each recorded training stage and relevant weight projection.

We then apply truncated SVD to the averaged residual and use the fixed projection matrices defined in Eq. (4) to construct target-compatible LoRA factors. In our Qwen implementation, target Q/K/V projections share an attention prior constructed from source Q residuals, while FFN up-projection (U) and down-projection (D) use their corresponding source projections. Thus, multiple target layers within the same segment share the prior derived from the corresponding source segment.

Concrete example. For Qwen2.5-0.5B → Qwen2.5-7B, the source model contains 24 Transformer layers and the target model contains 28 layers. Both are partitioned into eight depth-ordered segments, with the exact layer correspondence shown below (zero-based, inclusive indices).

SegmentSource LayersTarget Layers
00–20–3
13–54–6
26–87–9
39–1110–12
412–1413–15
515–1716–18
618–2019–21
721–2322–27

For instance, target layers 0–3 share the prior constructed by averaging the parameter residuals of source layers 0–2 at each recorded training stage, followed by truncated SVD and dimensional projection. The same procedure is applied to the remaining corresponding segments.

We will add these implementation details to the revised manuscript to improve reproducibility.

Reviewer_G71MOpenReview ↗

Keep the rating

Thanks for the information. I do not change the rating.

We sincerely thank the reviewer for their time and consideration. We appreciate the reviewer's positive evaluation of our work.

Comments on the paper

1 comment
Area_Chair_XHnhOpenReview ↗

Author Reviewer Discussion

Dear reviewers,

Thanks again for agreeing to review for ICLR.

As we are approaching the end of the discussion period, I would appreciate your acknowledgment of the authors' rebuttal and any feedback you would like to share with the authors.

This will help to improve the submissions, streamline decision making, and keep ICLR's original idea of having such a discussion period valuable to all participants in the review process.

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.