acceptodds
Under review as a conference paper at ICLR 2027

Rethinking State-Control Dynamics for Unified and Efficient Transformer Adaptation

Abstract

State-based fine-tuning reduces training memory by injecting lightweight controls into model states rather than modifying pretrained weights. However, these controls generally cannot be merged into the pretrained weights and must be evaluated at every discretization step, limiting inference throughput and practical deployment. Existing approaches incur two avoidable overheads: (1) they employ fine-grained subblock-level discretization, which requires frequent control evaluation, and (2) they instantiate separate controls across layers, forcing multiple control features to share a single routing during communication, limiting adaptation flexibility. In this work, we introduce **SURF** (**S**imilarity-Guided **U**nified **R**outing of **F**eatures), a unified state-based adaptation framework that integrates coarse-grained discretization with similarity-guided routing of shared control features. SURF composes consecutive residual subblocks into layer-wise transitions, reducing control evaluations without altering the pretrained transformations. It then groups layers with similar representations and applies sparse, token-dependent routing over a shared pool of rank-one control features, enabling feature reuse across depth while preserving input-specific adaptation. Experiments on 34 datasets and 10 pretrained models show that SURF substantially narrows the inference-efficiency gap with mergeable weight-based methods while matching or outperforming existing state-based approaches at comparable training cost. These results highlight discretization granularity and control sharing as two key design principles for efficient state-based adaptation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
State-based Fine-Tuning, Sparse Mixture of Experts, Control Theory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xrSiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SURF, a state-based Transformer adaptation framework combining similarity-guided layer grouping with sparse, token-dependent routing over shared rank-one features. The authors theoretically analyze coarse-control expressivity and feature reuse. Experiments across 34 datasets and 10 pretrained models demonstrate improved adaptation performance and inference efficiency compared with existing state-based methods.

Critical strengths

SURF addresses the inference overhead of non-mergeable state-based adapters, achieving decoding latency close to merged LoRA (1.05×) while substantially outperforming MoC in inference efficiency. The paper provides extensive evaluations across 34 datasets and 10 pretrained models covering language, vision, reasoning, and time-series tasks, with ablations supporting the effectiveness of similarity-guided grouping and feature-level routing.

Critical weaknesses

W1. Although Theorem 3.1 establishes the feasibility of coarse control under specific assumptions, it remains unclear whether SURF's fixed shared feature pool and sparse routing can realize the theoretical construction without quantified approximation errors.

W2. Figure 3 and Table 11 demonstrate end-to-end efficiency gains, but lack a breakdown separating intervention reduction, controller computation, activation movement, and routing overhead; the contribution of offline grouping costs is also unclear.

W3. Insufficient justification for similarity-guided grouping. Although cosine similarity measures representation similarity, its relationship with control-merging errors remains unverified, leaving unclear why it is an appropriate criterion for selecting intervention boundaries beyond empirical comparisons with random and uniform grouping.

W4. Lack of statistical robustness in key ablation results. The improvements in Tables 6 and 7 are relatively modest (0.7 and 0.5 percentage points over random grouping and MoC, respectively), but multi-seed results or confidence intervals are not reported.

W5. SURF estimates layer similarity using only 128 calibration examples per task, but the stability of grouping boundaries across calibration subsets and the criterion for selecting similarity thresholds remain insufficiently established.

W6. Incomplete controlled analysis of intervention frequency. Although Tables 6 and 14 and Figure 4(d) examine grouping effects, systematic comparisons across intervention counts under matched controller capacity and training settings are needed to disentangle intervention frequency from grouping locations and quantify accuracy–efficiency trade-offs.

Questions for authors

Please refer to Weaknesses W1–W6 for my questions and concerns.

3 comments

Thank you for the additional experiments and clarifications. However, I remain uncertain about why coarse-grained control achieves higher accuracy than fine-grained subblock-level control, beyond its efficiency advantages. Although Table 1 shows that layer-wise control outperforms subblock-level control under matched parameter budgets, the two settings use different per-intervention capacities. Moreover, the additional experiments in W6 show that increasing the number of interventions from 4 to 16 consistently improves accuracy, albeit at a higher computational cost.

I would therefore appreciate a more comprehensive comparison between coarse-grained and fine-grained control using the same controller architecture. In particular, if additional computational overhead or controller capacity is allowed, can fine-grained subblock-level control achieve higher accuracy than SURF?

Thank you very much for your detailed response and the additional experiments.

While the underlying reasons why fewer interventions can lead to higher accuracy remain somewhat unclear, I acknowledge that the additional experiments have substantially strengthened the empirical justification for the proposed design.

AuthorsOpenReview ↗

Thank You for Your Constructive Feedback

Dear Reviewer,

We sincerely thank you for your thoughtful follow-up and for taking the time to carefully consider our additional experiments and clarifications.

We are grateful that you have acknowledged the strengthened empirical evidence, and we greatly appreciate your constructive feedback throughout the discussion, which has helped us substantially improve our empirical analysis and clarify the scope and limitations of our work.

We have made every effort to address your concerns and sincerely hope that these improvements will support a more favorable assessment of our work.

We truly appreciate your time, thoughtful consideration, and valuable contribution to the ICLR 2027 review process!

Sincerely,

The Authors of Paper #19432

Reviewer_EY8qOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SURF to reduce the inference overhead of state-based Transformer adaptation. It groups consecutive layers using pretrained hidden-state similarity and applies one boundary intervention per group, with token-dependent routing over shared rank-one features. Experiments across a broad collection of datasets and pretrained models report competitive accuracy, reduced parameter budgets, and improved inference efficiency.

Critical strengths
  1. This efficiency improvement directly addresses a specific deployment challenge and is backed by measurable results.
  2. The empirical evidence demonstrates that the method has practical value extending beyond a single core or modality. Tables 2–5 show favorable overall performance across common-sense reasoning, mathematical reasoning, vision, and forecasting.
Critical weaknesses

1.The nonlinear theorem does not establish the capacity claim for the controller actually used by SURF. Equation (7) uses input-dependent Jacobian-transported directions, and Appendix C.1 explicitly acknowledges that its low-dimensional subspace is per-input, whereas Equation (9) uses a globally shared, fixed feature dictionary; neither their approximation relationship nor a connection from the similarity threshold to the error bound is established.

2.The evidence for the incremental accuracy advantage is weakened by unresolved selection and comparison details. Appendix E.2 specifies extensive SURF searches over thresholds, allocation schemes, routing budgets, and pool sizes without identifying the selection split.

Questions for authors

1.Does Theorem 3.1 claim approximation by the implemented Equation (9), or only by the Jacobian-based construction in Equation (7)? Given the input-dependent subspaces acknowledged in Appendix C.1, what result in the current paper connects that construction to a fixed shared dictionary and the adaptive feature-budget rule?

2.What exact training corpus, data splits, and checkpoint-selection protocol underlie Table 2, and which settings were verified to match the imported Control/MoC results?

2 comments
AuthorsOpenReview ↗

Follow-Up on Our Latest Response

Dear Reviewer EY8q,

Thank you again for your time and constructive feedback. We have carefully considered your comments and provided additional clarifications and experimental evidence in our latest responses and revised manuscript.

As the discussion period closes on November 18, Anywhere on Earth, we would be grateful if you could review these updates. Should any questions or concerns remain, we would be happy to address them before the deadline.

We hope the additional evidence addresses your concerns and may help inform your final assessment of our submission.

Sincerely,

The Authors of Paper #19432

Dear Reviewer EY8q,

We greatly appreciate your constructive feedback, which has helped us improve both the technical content and presentation of the paper.

As the Author–Reviewer Discussion period closes in few hours, on November 18 AoE, we would be grateful if you could review our latest responses and update your assessment if your remaining concerns have been adequately addressed.

We hope that the additional evidence and clarifications provide sufficient grounds for a favorable reconsideration.

Sincerely,

The Authors

Reviewer_jN6HOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces SURF (Similarity-Guided Unified Routing of Features), a state-based parameter-efficient fine-tuning (PEFT) framework. It addresses two overheads in existing state-based methods with two proposals:

  1. Coarse-grained control: Instead of editting feature of every layer, the authors group consecutive layers by pretrained hidden-state cosine similarity, applying one control per group boundary. This reduces the number of operations and reduces the runtime.
  2. Mixture-of-Features: A sparse token-dependent router selects rank-one features from a shared pool, enabling cross-depth feature reuse at a finer granularity than Mixture-of-Control (MoC)'s control-level routing. This improves the representation quality.

The authors provide theoretical analysis and evaluate on 34 datasets across 10 models (language, vision, and time-series).

Critical strengths
  • Comprehensive and empirical evaluation: Experiments span 34 datasets with three modalities (language, vision, and time-series). This supports the "general framework" claim.
  • Theoretical grounding tied to design: Theorem 3.1 motivates the adaptive feature allocation scheme, and Proposition 3.2 formalizes when feature reuse is necessary. The theory is modest but genuinely connected to the method and interesting.
Critical weaknesses
  • Inconsistency between the introduction and the experimental results: The introduction argues that weight-based PEFT methods must retain activation features for backpropagation, meaning that reducing the number of PEFT parameters does not necessarily reduce memory requirements. In contrast, state-based PEFT methods do not require retaining these activations, so reducing the number of parameters should reduce memory requirements. However, Table 2 shows that the memory requirements of SURF change very little as the number of parameters varies. Furthermore, SURF has memory requirements comparable to those of LoRA, a weight-based PEFT method.
Questions for authors
  • Memory Requirements: For vision tasks, SURF requires less memory than weight-based PEFT methods, as described in the introduction. However, this does not appear to be the case for language tasks. Why is this? Providing memory requirements for LoRA and SURF with varying numbers of parameters would make this claim more convincing.

  • Ablation Metrics: Since the trade-off between performance and efficiency is important for PEFT, it would be helpful to include efficiency-related metrics in Table 7 as well.

3 comments
AuthorsOpenReview ↗

Follow-Up on Our Latest Response

Dear Reviewer jN6H,

Thank you again for your time and constructive feedback. We have carefully considered your comments and provided additional clarifications and experimental evidence in our latest responses and revised manuscript.

As the discussion period closes on November 18, Anywhere on Earth, we would be grateful if you could review these updates. Should any questions or concerns remain, we would be happy to address them before the deadline.

We hope the additional evidence addresses your concerns and may help inform your final assessment of our submission.

Sincerely,

The Authors of Paper #19432

Reviewer_jN6HOpenReview ↗

Reply to Author

Thank you for conducting the additional experiments and evaluations. In particular, the ablation study, including inference speed and memory consumption, further demonstrated the effectiveness of the proposed method. I will maintain my positive score.

Dear Reviewer jN6H,

We sincerely thank you for your time, effort, and valuable feedback during the conference review process, and we are glad to know that our rebuttal addresses most of your concerns and enhanced the paper's overall clarity.

We sincerely appreciate your positive feedback and acknowledgment of the additional ablation, inference-speed, and memory-consumption analyses.

Best regards,

The Authors

Comments on the paper

1 comment

Dear reviewers Thank you for your thoughtful review. The authors have submitted their rebuttal, and the discussion period is now open. Please take a moment to read the authors' response and engage with them and your fellow reviewers to address any remaining questions or clarify key points. Your active participation during this phase is vital to reaching a fair and well-informed final recommendation.

Best regards, area chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.