acceptodds
Under review as a conference paper at ICLR 2027

ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning

Abstract

Popular low-rank parameter-efficient fine-tuning (PEFT) methods represent adaptation as separate updates to selected backbone weights, without maintaining an explicit task-specific state that is updated and reused across depth. These updates also require the backbone at inference and therefore cannot operate as standalone predictors. We propose ShadowPEFT, a stateful, detachable alternative to weight-space adaptation that consolidates trainable adaptation into a modular shadow component centered on a compact shadow model and lightweight Transformer layer-specific coupling modules. A persistent shadow state refines the frozen backbone representations and is updated from them in an interactive manner. Because the shadow model is trained as a complete predictor, it can be detached for shadow-only inference without executing the base model and can be initialized from a pretrained model. Experiments on text and image generation and understanding benchmarks show that ShadowPEFT matches or outperforms LoRA and DoRA under comparable trainable-parameter budgets. Additional analyses on shadow pretraining, cross-dataset transfer, parameter scaling, inference latency, and system-level evaluation suggest that centralized layer-space adaptation is a competitive and flexible alternative to conventional low-rank PEFT.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
parameter-efficient fine-tuning, large language models, adaption, peft

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_A5mCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose ShadowPEFT, a novel PEFT method that couples the frozen backbone to a compact shadow network via layer-wise bottleneck injection and gated residual updates. The shadow design supports both attached refinement and shadow-only inference through separate training, enabling edge-cloud collaboration. They empirically evaluate their method against LoRA, DoRA, and other PEFT baseline methods on language benchmarks, DreamBooth, and a robotic routing task, showing competitive accuracy.

Critical strengths
  1. The paper is well written and is easy to follow
  2. The detachable design of the adapter can be attached to the full model to improve accuracy or detached as a standalone edge predictor
  3. Their method consistently achieves better performance than the baselines at lower parameter counts
  4. The ablation study highlights the importance of the inject and update modules and shows that both are required for improved accuracy.
Critical weaknesses
  1. Unlike LoRA, where the weights can be merged into the backbone model for zero inference latency and memory overhead, ShadowPEFT cannot be merged with the backbone
  2. Table 11 reports the throughput and memory usage for a single synthetic optimizer step, but the total wall-clock training time for the actual benchmarked models is missing.
  3. For budget-matched setups, detached shadow performance completely collapses (such as 1.9% on MMLU) unless initialized with a heavy 455M pretrained checkpoint that destroys parameter efficiency.
Questions for authors
  1. What is the actual wall-clock fine-tuning time for all the methods on the main benchmarks?
  2. Can you show the decoding throughput and peak KV-Cache memory for the 4B and 8B models?
Reviewer_tYUGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ShadowPEFT, a parameter-efficient fine-tuning framework in which a small shadow model serves both as an adaptation component for a frozen large model and as a standalone predictor. The framework combines a persistent shadow state with bidirectional interaction between the shadow and backbone, and jointly optimizes the attached model and the shadow's own predictions. This design supports both high-capacity attached inference and lightweight shadow-only inference. Experiments evaluate adaptation performance across language and image-generation tasks, standalone shadow performance, and a two-tier edge–cloud deployment scenario.

Critical strengths

1. An interesting and relatively novel formulation that combines adaptation and standalone prediction.

The idea of training the same small model to serve as both an adapter for a frozen backbone and an independently deployable predictor is a distinctive aspect of the paper. This formulation provides a potentially useful connection between parameter-efficient adaptation and deployment at different computational budgets. In particular, obtaining two inference modes through a shared task-training process could be valuable for applications requiring both a capable server-side model and a lightweight local predictor. The paper deserves credit for exploring this joint formulation and demonstrating its feasibility.

2. Competitive adaptation performance across multiple tasks and backbone scales.

The experiments provide substantial evidence that ShadowPEFT is an effective adaptation method. Under approximately matched trainable-parameter budgets, it achieves the highest average score across all three Qwen3 scales and the best result in 14 of the 15 task–scale combinations in Table 1. The additional comparisons with Llama 3.2 and a broader set of PEFT baselines strengthen this finding. Although the improvements are generally modest, their consistency across tasks and model scales supports the practical viability of the proposed architecture.

Critical weaknesses

1. The benefit of jointly realizing adaptation and standalone prediction remains insufficiently established.

The central idea is appealing, but the paper does not sufficiently explain or demonstrate what is gained by assigning both roles to the same jointly trained model. The limitations identified for conventional PEFT motivate exploring a different architecture; establishing the value of this particular joint formulation also requires comparison with independently adapting a large model and fine-tuning a small predictor.

The current results establish competitive attached performance and demonstrate standalone prediction under suitable configurations. However, useful detached generation performance is obtained in a separate high-capacity setting initialized from Qwen3-0.6B, with additional alignment pretraining. The compact configurations used in the main matched-budget comparisons have weak standalone generation performance. The paper acknowledges this dependence on pretrained capability.

Consequently, the experiments leave unclear whether joint training improves the small model relative to independent fine-tuning, whether retaining the standalone prediction objective benefits large-model adaptation, or whether the resulting pair offers a better overall performance–cost trade-off. A clear advantage along any of these dimensions would substantially strengthen the significance of the central design.

2. The practical advantages of the shared training and deployment framework receive only preliminary support.

The claim that both inference modes can be obtained without separate task-training runs is attractive, but the paper does not establish the total cost of obtaining both usable models relative to an independently trained pair. The training-cost analysis in Appendix J evaluates compact PEFT configurations, while the useful detached generation and edge–cloud results use the separate high-capacity pretrained-shadow configuration. This leaves the end-to-end efficiency of the proposed dual-purpose solution unresolved.

The edge–cloud experiment does include an appropriate conventional cascade, and ShadowPEFT obtains a favorable result. However, the evaluation contains 324 requests and a single run, with three false actions versus four for the conventional cascade and total decoding time of 84.8 versus 93.5 seconds. The larger advertised speedups are measured against cloud-only systems, and the paper itself attributes much of this benefit to two-tier routing. These results provide encouraging application-level evidence, while leaving the broader advantage of the joint formulation insufficiently established.

Questions for authors

1. Benefits to prediction quality.
Can the authors provide a controlled comparison between joint ShadowPEFT training and independently training a small predictor and a PEFT-adapted large model, using the same initial models and task data? For the small predictor, starting from the same aligned shadow checkpoint would help isolate the contribution of joint training. Reporting both standalone-small and adapted-large performance would clarify whether the proposed formulation offers complementary gains or introduces a trade-off between its two roles. An additional comparison with adaptation-only training would help establish whether the standalone prediction objective benefits attached performance.

2. Benefits to total training cost.
Can the authors compare the end-to-end training cost of obtaining both deployable models through ShadowPEFT with that of obtaining them through independent training, including the shadow's additional alignment pretraining? Reporting total GPU-hours and the resulting performance of both models would substantiate the practical value of obtaining two inference modes through one shared task-training process.

3. Benefits in the intended deployment scenario.
Can the authors strengthen the existing comparison with the conventional two-tier cascade through repeated runs or additional workloads, and report the resulting accuracy–latency trade-offs? In particular, evaluating different proportions of requests requiring the large model would help establish when the shared ShadowPEFT design provides a meaningful deployment advantage.

Reviewer_y3qvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposes ShadowPEFT, a stateful, detachable alternative to weight-space adaptation that consolidates trainable adaptation into a modular shadow component centered on a compact shadow model and lightweight Transformer layer-specific coupling modules. In experiments, it shows much faster.

Critical strengths
  1. A persistent cross-layer adaptation state methods, it is quite different from traditional weight adaption approaches like LoRA etc. The detachable design of the PEFT methods opens up a new direction to be practically used for edge devices.

  2. Quite several baselines and backbones. The comparison covers LoRA, DoRA, DeLoRA, LoHA, LoKr and LST across two model families.

Critical weaknesses
  1. Experimental results gain is marginal. Average margins over the best baseline are tiny: +0.19 at 0.6B, +0.58 at 4B and +0.45 at 8B. Not sure if it comes from random seed. it is unclear to me how accumulated shadow states from different layers actually aid in attaining such task-relevant information.

  2. The significance test is unexplained. Table 1 gives no standard deviations. The claim of “p < 5% in a paired t-test” does not say what is paired: runs, tasks or examples. A p-value from 5 cells of averages would not mean much.

  3. Auxiliary networks and representation-level adaptations have been explored in works like Side-Tuning (Zhang et al., ECCV 2020), LST (Sung et al., NeurIPS 2022) and ReFT (Wu et al., NeurIPS 2024); And the gated updates are closely related to conventional recurrent mechanisms. While the usage of bidirectional interaction is different, the clear contribution of the proposed method with the prior methods should be demonstrated.

Questions for authors

Please answer all my concerns in weakness.

  1. How does persistent shadow state make the proposed method more effective over prior side-network and representation-level methods like LST or ReFT?
  2. Could the authors provide further analysis on how information accumulated across layers makes it more task-relevant and/or compare against a simpler state update mechanism under a matched budget?
  3. A clear statistical significance analysis for the gains in Table 5, and a clarification on whether the degradation of LoRA and DoRA under larger parameter budgets still holds after independently optimizing hyperparameters at each parameter budget, is needed to ensure that the proposed claim stands.
  4. Given the claim that the proposed method to be more effective, can the authors clarify how much of the efficiency improvement comes from ShadowPEFT, rather than the edge–cloud routing?
2 comments
AuthorsOpenReview ↗

Response to Reviewer y3qv [1/2]

We would like to thank Reviewer y3qv for your helpful comments. Below are our answers to your concerns.

W1: Experimental results gain is marginal.

Answer:

We agree that the per-cell margins are modest. We would argue that this is characteristic of progress in PEFT, not a weakness specific to ShadowPEFT. DoRA, a widely used LoRA variant, improves over LoRA by only +0.27/+0.30 average points at 0.6B/4B and falls 0.52 points below LoRA at 8B (Table 1); even its positive gains are smaller than ShadowPEFT's relative improvement.

Against that backdrop, we restate the contribution. ShadowPEFT is not a new branch of the LoRA family, and we do not claim state-of-the-art accuracy. It is a different PEFT paradigm: the trainable adaptation is concentrated in a shadow network that is itself a complete predictor. A single training run therefore supports two deployment modes: attached to the backbone, or detached as a standalone model.

The margins are not from seed artifacts: we ran extensive experiments without selecting favorable seeds, and the improvement direction is consistent throughout (best average at every scale, 14/15 cells, all OOD groups; see also W2). Moreover, the gains trace to the shadow pathway rather than chance: removing the state update at matched budget reduces accuracy on both task types (Table 3).


W2: The significance test is unexplained. Table 1 gives no standard deviations. The claim of “p < 5% in a paired t-test” does not say what is paired: runs, tasks or examples. A p-value from 5 cells of averages would not mean much.

Answer: Thanks for pointing out this issue. The reported p-value in Table 1 comes from a paired t-test between ShadowPEFT and DoRA computed on the five-run average scores; it was not conducted per task, and the caption did not state this. We will add it in the camera-ready version.


W3 and Q1: How does persistent shadow state make the proposed method more effective over prior side-network and representation-level methods like LST or ReFT?

Answer:

We agree that these families exist, and we have discussed the differences between LST and ReFT in Section 2. The distinction we claim is not the presence of a side network or a gating function. It is the combination of three properties:

  • (i) a persistent, detachable state carried across depth;
  • (ii) bidirectional per-block coupling — shadow→base injection and base→shadow update;
  • (iii) a complete standalone predictor trained jointly with the frozen backbone.

Compared with LST. LST is compared directly in Table 2 at matched budgets and underperforms both ShadowPEFT and LoRA substantially. Its ladder consumes frozen features in one direction; ShadowPEFT's coupling is bidirectional, and its shadow is jointly supervised as a standalone predictor.

Compared with ReFT. ReFT-style interventions are layer-local and independent across depth: successive interventions cannot accumulate task information through a shared state. ShadowPEFT's injections, by contrast, are connected by the evolving shadow state through the update module.

Compared with recurrent mechanisms. We acknowledge that the gated update is GRU-like (cited as Cho et al., 2014, in Section 3.3) and we do not claim novelty for the gate itself. Novelty is at the method level, not the component level: gated recurrent updates are an established building block. The method-level contribution here is the persistent cross-layer state together with its detachable, jointly trained predictor — properties (i)–(iii) above, which the LST comparison and the Table 3 ablation demonstrate empirically.


Q2: A clear statistical significance analysis for the gains in Table 5.

Answer:

Thanks for your suggestions. All methods in Table 5 follow the same evaluation protocol:

  • Model selection. For each of LoRA, DoRA, and ShadowPEFT, we take their best-performing checkpoint of five runs from the main experiment (Table 1).
  • Inference. All methods are evaluated with a fixed random seed of 42 for a fair comparison. Seed 42 is a conventional default, not one chosen to favor any method.

With both the trained checkpoint and the seed fixed, inference is almost deterministic: rerunning the evaluation reproduces nearly identical scores. There is no run-to-run variation left to estimate, so we did not attach a significance test to Table 5. We will state this protocol explicitly in the Table 5 caption in the final version.

AuthorsOpenReview ↗

Response to Reviewer y3qv [2/2]

Q3: Given the claim that the proposed method to be more effective, can the authors clarify how much of the efficiency improvement comes from ShadowPEFT, rather than the edge–cloud routing?

Answer:

We have studied the efficiency of ShadowPEFT itself thoroughly in Section 4.5 and Appendix J. During the discussion period, we added Appendix K (Table 13), which extends the efficiency study to the 4B and 8B backbones. All runs use the same protocol as Table 4.

We draw two observations. First, ShadowPEFT becomes more efficient as the backbone grows: compared with LoRA, it is slower on Qwen3 0.6B (Table 4) but faster on Qwen3 4B and 8B (Table 13). Second, peak memory is on par with LoRA, within 0.3% (15,756 vs. 15,796 MiB at 8B), and the ShadowPEFT-vs-LoRA gap changes by at most 1.8 percentage points when the KV cache is disabled (under 0.6 on decoding), so the cache is not what drives the trend.

The scale trend reflects where the two methods place their extra computation. LoRA attaches a low-rank adapter to each of several linear layers inside every transformer decoder layer, so its overhead accumulates with both depth and width. ShadowPEFT instead adds a much smaller injection/update coupling per decoder layer, together with a single shadow decoder layer whose internal widths are fixed and do not grow with depth; that layer is wider than the backbone itself on the smallest model and far narrower on the largest. As depth and width grow, the extra compute LoRA adds in every decoder layer accumulates over more and wider blocks, while ShadowPEFT's single added layer accounts for less of the total. The ranking therefore flips between 0.6B and 4B.

On the edge–cloud system in Appendix F: the paper already states that the reported speed-ups come from two-tier routing itself. ShadowPEFT's contribution there is that the edge model comes out of the same training run at no additional cost.

Taken together, ShadowPEFT matches LoRA on smaller backbones across all measured stages, training, attached inference, and standalone inference, regardless of the routing scenario, and may outperform it on larger ones, such as 8B.


Thank you again for your very kind effort and professional input! We hope Reviewer y3qv could consider raising the score accordingly if we have addressed your concerns. If you have any further concerns, please don't hesitate to let us know.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Looking forward to your reply

Dear Reviewer y3qv, tYUG, A5mC,

Thank you for your constructive comments on ShadowPEFT. We have replied point by point.

With the discussion period ending shortly, we would be grateful for any indication of whether our replies have addressed your concerns. If any point remains open, we would be glad to clarify it or run additional experiments while time allows.

We appreciate the time and care you have already given our submission.

AuthorsOpenReview ↗

Looking forward to your reply

Dear Reviewers,

Thank you again for the care and rigor you brought to your reviews of ShadowPEFT. We know this is a busy stretch of the discussion period, and we're grateful for the time you have already given our submission.

We've posted point-by-point responses. Whenever it is convenient, we'd be grateful for even a brief note on whether those replies have resolved your concerns. If so, we hope the updated manuscript and responses make the contribution clearer in your final assessment.​ If anything remains open, we're glad to clarify it or run further experiments while time allows.

Thank you for your time, and for helping us make the work better.

Warm regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.