- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper studies how to maintain a task-specific model when its base model changes. It compares keeping the old model, copying its adapter, distilling its predictions, and retraining. Experiments cover four Qwen generations and selected OLMo checkpoints. The paper also proposes a policy for choosing when to update and reports lower training cost.
Critical strengths
-
The comparison across model versions addresses a practical problem. It measures task quality, prediction changes, and update cost together. This gives a useful picture of the choices available when a base model is replaced.
-
The OLMo tests control for template, precision, and output format. They show that matching architectures alone do not guarantee successful adapter transfer.
Critical weaknesses
-
The cost claim depends on how the reference model is obtained. Algorithm 1 uses the fully retrained target model's score to decide whether retraining can be skipped. Table 4 reports 15.6 versus 47.7 GPU-hours, but it is unclear whether reference training and rejected candidates are included. Without these costs, the saving may be overstated.
-
The broad claim about weights and data goes beyond the tested baselines. The paper mainly tests direct adapter copying, without alignment methods such as Cross-LoRA [1]. Trans-LoRA [2] already studies transfer through distillation. The refresh experiments also cover only one Qwen upgrade, which limits the general conclusion.
Questions for authors
-
How is the reference score obtained, and does Table 4 include all costs needed to make the update decision?
-
Does an alignment-based transfer method change the result on a model pair where it applies?
References
[1] Xia et al. Cross-LoRA: A Data-Free LoRA Transfer Framework across Heterogeneous LLMs. 2025.
[2] Wang et al. Trans-LoRA: towards data-free Transferable Parameter Efficient Finetuning. 2024.
2 comments
Thanks for the clear response and revision. Since the main soundness error is now fixed, I am raising my rating from 1 to 2.
Two concerns remain (also noted by Reviewers vRfP and KMAA):
- With the cost-saving claim withdrawn, the one-hop coupling forecast negative, and REFRESH limited to teacher-seen inputs, the remaining result that naive copying fails across independent runs while freezing, retraining, or distilling works is largely expected.
- No alignment or learned transfer baseline was tested, so it remains open how much of the porting gap existing transfer methods would close.
Overall, I think a rating of 2 accurately reflects the quality of the current manuscript.
Q1. How is the reference score obtained, and does Table 4 include all the costs needed to make the update decision?
Your reading of Algorithm 1 is correct, and Table 4 did not include those costs. The 15.6 GPU-hours counted only the actions that were finally selected. Evaluating the decision branches requires the retrained reference in all 33 episodes; one episode also reaches the REFRESH test, and that candidate was selected. FREEZE and PORT candidates need no training, so there are no other rejected candidates with a training cost. Charging the references once each, plus that student, gives 50.7 GPU-hours and 33 gold-label episodes, against 47.7 and 33 for always retraining. These are migration-training costs only; scoring, serving and data management were outside the accounting in both columns. The cost-saving claim is withdrawn.
Q2. Does an alignment-based transfer method change the result on a model pair where it applies? The refresh experiments cover only one Qwen upgrade.
We have no matched Cross-LoRA or Trans-LoRA result, so whether alignment changes the picture on these pairs is an open question that we will state as such. One clarification we owe you: REFRESH follows the distillation route that Trans-LoRA proposed, as §5.4 says; it differs in distilling on retained real inputs, whereas Trans-LoRA is near-data-free. We will make that relation explicit in Related Work. The failure in §5.3 is specific to copying adapters directly across the independent runs we measured. The REFRESH results come from one Qwen upgrade (CLINC150 extends it to a second task class on the same hop) and from teacher-seen inputs, so they cannot establish a general ordering of weights, data and learned transfer methods, and we will narrow the claim accordingly.
Changes 1, 3, 6 and 8 cover this review.