- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes AR-Retrofit, a parameter-efficient framework that adds cross-depth representation aggregation to frozen pretrained decoders through gated residual corrections and token-dependent strength modulation. It further introduces ReSkip, which reuses learned routing signals and offline-calibrated thresholds to selectively bypass layer computations without training an additional controller. Experiments on language and vision-language benchmarks and downstream LIBERO policies demonstrate improved average task performance, with ReSkip achieving 4.5%–8.8% higher VLM decoding throughput than the original Qwen3-VL models while retaining most adaptation gains.
Critical strengths
This work propose a novel adaptation methods that introduce attention residuals, which I think promising. AR-Retrofit introduces cross-depth aggregation while keeping the pretrained backbone frozen and adding few trainable parameters. Its zero-initialized gating schedule preserves the original forward function at initialization, providing a principled starting point for adaptation.
Critical weaknesses
-
The evaluation in this paper is insufficient. The proposed method is essentially a form of PEFT (Parameter-Efficient Fine-Tuning) or adaptation methods, such as various LoRA variants or other adapter methods. However, in the paper’s main results (e.g., Table 1), the authors compare their method only against the "Base" model to report the improvement. The "Base" is not a appropriate baseline. Since the "Ours" involves downstream fine-tuning built upon that "Base" model, such an improvement is to be expected and does not effectively demonstrate the specific advantages of the proposed method. A more rigorous evaluation would involve comparing the proposed method against other existing PEFT/adaptation methods. Although the authors provide such a comparison in Table 4, it remains insufficient, as Table 4 covers only a single model. The significant drop in Block Adapter performance (from 71.93 to 59.40) is left unexplained, which could reduce the confidence of the results.
-
The calibration risk proxy requires further validation, and the threshold-selection procedure is not fully specified. For example, the rationale for "plus 0.25 if any omission changes the highest probability token" are also unclear. Although the final thresholds are reported, the candidate quantile levels, admissible coverage intervals, and precise definition of the coverage-deviation term could be insufficiently specified to reproduce the calibration procedure.
Questions for authors
Please refer to the Critical Weaknesses. The primary issue of this work is that compared baselines are inadequate. The most appropriate baselines would be other PEFT/adaptation methods, rather than the "base" model itself.
Response Summary by Authors
We thank all reviewers for their constructive feedback. We have incorporated the results of the discussion into the revised paper.
Strengths noted by the reviewers
After the discussion, Reviewer wWbB raised the rating to Weak Accept, and Reviewer No7b raised the rating and has no further concerns. We have addressed all questions raised by Reviewer qWzt.
Main concerns and our responses
Comparison with PEFT under matched supervision (all reviewers). LoRA and DoRA now use the same teacher KL, training data and steps as AR-Retrofit (ARR), with a matched parameter budget, and ARR has the highest VLM and LLM averages on both Qwen3-VL-2B and 4B. This comparison is now in the main text as Table 1 (Section 4.2).
Low score of Block Adapter (all reviewers). Block Adapter is a local baseline without cross-depth differences, so its correction also acts on the text embeddings and is much larger than the input. With only ARR's zero correction at the first block, its score recovers, which shows that the cross-depth difference design is effective (Appendix A10.4).
Design of the components (qWzt, wWbB). Under the full ARR objective and over four training runs, a variant that replaces the adapter and RSM with a learned scalar scores below ARR in every run. Dynamic routing and token-dependent RSM further raise the average, and they provide the per-token signal that ReSkip reads (Section 4.5, Appendix A10.5).
ReSkip decisions beyond layer selection (qWzt, wWbB). With the same candidate layers, trigger frequencies and nearly the same latency, ReSkip scores above fixed and random schedules. Positions with a high most recent source weight have a lower omission KL, which shows that this weight marks where skipping is safe (Appendices A10.3 and A10.6).
Speed at the actual output length (qWzt, wWbB). Timed at the actual length of each answer and including the prefill cost of ARR, ReSkip is faster than ARR on all six VLM tasks and also has higher throughput than Base. We also specified how
acc_normis scored when only decoding skips layers (Appendices A10.1 and A10.2).We thank the reviewers and the Area Chair again for their time and effort.