acceptodds
Under review as a conference paper at ICLR 2027

AR-Retrofit: Retrofitting Pretrained Decoders with Attention Residuals for Adaptive Depth

Abstract

Attention Residuals improve Transformer performance by enabling adaptive aggregation of representations across different depths. However, applying this mechanism to existing pretrained models remains challenging, as direct integration requires expensive backbone retraining and introduces additional inference overhead. To avoid costly backbone retraining, we propose Attention Residual Retrofit (AR-Retrofit), a lightweight framework for frozen pretrained decoders. AR-Retrofit introduces depth aggregation through gated residual corrections while preserving the original forward function at initialization. The magnitude of these corrections is adaptively controlled by Residual Strength Modulation for different tokens and depths. To reduce the additional inference overhead, we propose Routing-guided Residual Skipping (ReSkip), which reuses the routing signals learned by AR-Retrofit for selective layer execution without training an additional controller. Experiments on language, multimodal, and robotic tasks show that AR-Retrofit transfers adaptive depth aggregation to pretrained models while improving performance across all three domains. On Qwen3-VL-2B, it adds less than 0.4% trainable parameters and requires only 12.6 minutes of training on a single H100. Across six VLM benchmarks, average scores improve by 2.08 and 1.44 points for Qwen3-VL-2B and 4B, respectively. ReSkip increases decoding throughput over Base by 4.5%–8.8% while retaining most gains. These results show that our framework enables lightweight and efficient adaptive depth in pretrained models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Attention residuals, Model retrofitting, Parameter efficient adaptation, Adaptive computation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_No7bOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AR-Retrofit, a parameter-efficient framework that adds cross-depth representation aggregation to frozen pretrained decoders through gated residual corrections and token-dependent strength modulation. It further introduces ReSkip, which reuses learned routing signals and offline-calibrated thresholds to selectively bypass layer computations without training an additional controller. Experiments on language and vision-language benchmarks and downstream LIBERO policies demonstrate improved average task performance, with ReSkip achieving 4.5%–8.8% higher VLM decoding throughput than the original Qwen3-VL models while retaining most adaptation gains.

Critical strengths

This work propose a novel adaptation methods that introduce attention residuals, which I think promising. AR-Retrofit introduces cross-depth aggregation while keeping the pretrained backbone frozen and adding few trainable parameters. Its zero-initialized gating schedule preserves the original forward function at initialization, providing a principled starting point for adaptation.

Critical weaknesses
  1. The evaluation in this paper is insufficient. The proposed method is essentially a form of PEFT (Parameter-Efficient Fine-Tuning) or adaptation methods, such as various LoRA variants or other adapter methods. However, in the paper’s main results (e.g., Table 1), the authors compare their method only against the "Base" model to report the improvement. The "Base" is not a appropriate baseline. Since the "Ours" involves downstream fine-tuning built upon that "Base" model, such an improvement is to be expected and does not effectively demonstrate the specific advantages of the proposed method. A more rigorous evaluation would involve comparing the proposed method against other existing PEFT/adaptation methods. Although the authors provide such a comparison in Table 4, it remains insufficient, as Table 4 covers only a single model. The significant drop in Block Adapter performance (from 71.93 to 59.40) is left unexplained, which could reduce the confidence of the results.

  2. The calibration risk proxy requires further validation, and the threshold-selection procedure is not fully specified. For example, the rationale for "plus 0.25 if any omission changes the highest probability token" are also unclear. Although the final thresholds are reported, the candidate quantile levels, admissible coverage intervals, and precise definition of the coverage-deviation term could be insufficiently specified to reproduce the calibration procedure.

Questions for authors

Please refer to the Critical Weaknesses. The primary issue of this work is that compared baselines are inadequate. The most appropriate baselines would be other PEFT/adaptation methods, rather than the "base" model itself.

Reviewer_wWbBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AR-Retrofit, a method for adapting a frozen pre-trained LLM to use attention residuals. In addition to the standard AttnRes operation that yields an input-dependent summary of preceding block states, the difference between the summary and the preceding block state is transformed by an MLP adapter and injected into the residual stream through fixed scheduled and input-dependent scalar gates. The authors additionally propose ReSkip, which uses offline statistics of the input dependent residual weights and gates to select thresholds and candidate layers to selectively bypass accordingly, while retaining KV projections computation and caching. Experiments report performance gains across multiple benchmarks covering language and multimodal domains and downstream robotic policy training. ReSkip enables modest throughput gains on non-textual domains while retaining most gains.

Critical strengths

Whether adapting frozen LLMs to use attention residuals brings benefits over standard parameter/compute-matched tuning methods is an interesting and practically relevant question. The reported aggregate gains relative to the frozen model seem meaningful and consistent, and preliminarily (see Weaknesses for why the qualifier) also seem encouraging compared to standard PEFT. Leveraging the routing signals from the learned attention residuals for dynamic depth allocation is also an interesting idea.

Critical weaknesses
  • The specifics of the proposed architecture are not sufficiently justified. Preserving the pretrained function at initialization does not necessarily imply the use of the bottleneck adapter or RSM. One could use a scalar-gated interpolation between the previous state and the routed mixture, as equation 5 shows. The authors should also demonstrate the benefit of dynamic routing over fixed learned weights for the mixture, and of token-dependent RSM over learned constant gains per block. The RSM-strength ablation gives some signal, but does not truly resolve these questions, and should ideally report variability across training seeds.
  • The comparison against the LoRA and DoRA baselines has the objective as confounder. ARR is trained with teacher KL regularization, whereas the baselines use max. likelihood alone. Therefore, it is unclear if the architecture truly outperforms baselines with equivalent supervision.
  • It is not obvious to me why a high weight on a preceding block state indicates that the upcoming block is a good candidate to be skipped. The ReSkip ablation compares against randomly skipping layers, but to justify ReSkip it should be ablated against static skipping and random decisions restricted to the same candidate set. Otherwise it conflates skipping relatively safe layers as per the computed sensitivities with the input-dependent ReSkip rule. The inference gains of ReSkip are also questionable. VLM timing uses 64 output tokens, while the quality evaluation aggregates approximately four cached decoding steps per sample. Since prefill is not accelerated by ReSkip, and accordingly incurs an overhead due to the AttnRes parameters, these measurements do not truly establish comparable savings.
Questions for authors
  • What is the motivation for the bottleneck adapter and RSM? Why not simpler strategies like learned scalar gates? Is input-dependent mixing truly necessary or would a static learned mixture of layers perform comparably? Can you justify this with an ablation against truly comparable baselines (i.e. accounting for the objective confounder)?
  • How does ReSkip compare with static skipping and skipping random layers from the same set of candidate layers at matched bypass budgets?
  • What are the end-to-end inference throughput gains when measured using the actual generated lengths of the benchmark responses?
  • The score for the block adapter baseline in Tables 4 and A8-1 looks significantly worse than alternatives. Why is that? It is not clear to me what is the function that adapter implements.

Edit: Updated rating to Weak Accept

Reviewer_qWztOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AR-Retrofit, which adds cross-depth aggregation to frozen pretrained decoders through gated, low-rank nonlinear corrections at block boundaries. A zero-initialized gate schedule preserves the original forward function, while Residual Strength Modulation adjusts correction magnitude. ReSkip subsequently uses the learned routing signals and offline calibration to omit selected layer computations during cached decoding, retaining K/V updates and full prefill. ARR improves average text and VLM performance across four small backbones and improves initialization for subsequently trained LIBERO policies. On Qwen3-VL-2B/4B VLM tasks, ReSkip retains most of the quality gains while increasing measured decode throughput over Base by approximately 8.8%/4.5%.

Critical strengths
  1. The retrofit is simple and compatible with frozen decoders. Equations 2-6 provide a concrete mechanism for introducing historical block information without abruptly replacing the pretrained residual path. Its small parameter count, short adaptation runs and positive averages across several backbones support practical interest.
  2. The paper measures both quality and actual runtime. It specifies the retained K/V operations, calibration set and frozen policies, includes parameter/data/step-matched adaptation comparisons, and reports stage timings and paired timing intervals. The weaker LLM and VLA speed results are acknowledged rather than hidden.
Critical weaknesses
  1. The comparisons leave adaptive-component attribution unresolved. ARR uses teacher KL in addition to task loss, while A8.1 states that LoRA/DoRA use SFT alone. Thus Table 4 does not isolate the gain from cross-depth routing. The independently trained Block Adapter is a relevant baseline, but its large degradation needs initialization, gate and objective details. For ReSkip, Global Random samples all 36 layers, whereas the method uses sensitivity-selected subsets. This conflates selecting safe layers with making useful token-dependent decisions. A matched static or shuffled policy over the same candidates is needed to substantiate that part of the contribution.
  2. The evaluated scoring and timing paths need reconciliation. ReSkip applies only during cached decoding, yet A3 uses acc_norm on multiple-choice language tasks and A4 reports changed scores. Sequential cached scoring could explain this, but the implementation is unspecified. Separately, timing uses 64 generated VLM tokens while A9 shows mostly one or two cached steps in several sampled choice tasks. Since the first token uses full prefill, retained short-answer quality and fixed-length decode speed do not by themselves establish the same tradeoff for naturally terminated answers or longer reasoning. A5's total timings are useful but still reflect the fixed workload.
Questions for authors
  1. What happens with matched teacher-KL LoRA/DoRA and block-adapter training, and with a learned static depth mixture under the ARR objective? Please clarify the block adapter's initialization and gate settings.
  2. Does ReSkip outperform a static/shuffled policy with identical candidate layers and per-block trigger frequencies at matched latency and quality?
  3. How are acc_norm scores computed with decode-only skipping, and which scored tokens undergo skips? Please also report natural-stopping time-to-answer and clarify how often short VLM decisions are fixed by the first full-prefill token.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response Summary by Authors

We thank all reviewers for their constructive feedback. We have incorporated the results of the discussion into the revised paper.

Strengths noted by the reviewers

  • A novel and promising way to bring attention residuals into frozen pretrained models, which preserves the pretrained function at initialization (No7b, qWzt).
  • Few trainable parameters, short adaptation runs and consistent gains across backbones (qWzt, wWbB).
  • Reusing the routing signal for dynamic depth is an interesting idea (wWbB), and the paper measures both quality and actual runtime (qWzt).

After the discussion, Reviewer wWbB raised the rating to Weak Accept, and Reviewer No7b raised the rating and has no further concerns. We have addressed all questions raised by Reviewer qWzt.

Main concerns and our responses

  1. Comparison with PEFT under matched supervision (all reviewers). LoRA and DoRA now use the same teacher KL, training data and steps as AR-Retrofit (ARR), with a matched parameter budget, and ARR has the highest VLM and LLM averages on both Qwen3-VL-2B and 4B. This comparison is now in the main text as Table 1 (Section 4.2).

  2. Low score of Block Adapter (all reviewers). Block Adapter is a local baseline without cross-depth differences, so its correction also acts on the text embeddings and is much larger than the input. With only ARR's zero correction at the first block, its score recovers, which shows that the cross-depth difference design is effective (Appendix A10.4).

  3. Design of the components (qWzt, wWbB). Under the full ARR objective and over four training runs, a variant that replaces the adapter and RSM with a learned scalar scores below ARR in every run. Dynamic routing and token-dependent RSM further raise the average, and they provide the per-token signal that ReSkip reads (Section 4.5, Appendix A10.5).

  4. ReSkip decisions beyond layer selection (qWzt, wWbB). With the same candidate layers, trigger frequencies and nearly the same latency, ReSkip scores above fixed and random schedules. Positions with a high most recent source weight have a lower omission KL, which shows that this weight marks where skipping is safe (Appendices A10.3 and A10.6).

  5. Speed at the actual output length (qWzt, wWbB). Timed at the actual length of each answer and including the prefill cost of ARR, ReSkip is faster than ARR on all six VLM tasks and also has higher throughput than Base. We also specified how acc_norm is scored when only decoding skips layers (Appendices A10.1 and A10.2).

We thank the reviewers and the Area Chair again for their time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.